Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary
Since you have mentioned that, I'm kind of interested, who else is going to benefit from this transition? I'm asking because everyone tells me that it's essential for us to work together, though my senses tells me that we are starting from different prerequisites, targeting different use cases and taking the entirely different approaches to solve the problem. Actually, I'd say we have more differences than the things we have in common. Did you hear the same requests from anyone of your current users? May be we should invite them for discussion, just to make sure we would not have to redo it again. -- Kirill Izotov вторник, 15 апреля 2014 г. в 11:13, Joshua Harlow написал: Sure, its not the fully complete lazy_engine, but piece by piece we can get there. Of course code/contributions are welcome, as such things will benefit more than just mistral, but openstack as a whole :-) -Josh From: Kirill Izotov enyk...@stackstorm.com (mailto:enyk...@stackstorm.com) Date: Monday, April 14, 2014 at 9:02 PM To: Joshua Harlow harlo...@yahoo-inc.com (mailto:harlo...@yahoo-inc.com) Cc: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org (mailto:openstack-dev@lists.openstack.org) Subject: Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary Thank for pointing that out, Joshua. I had a look on [1] and it seems to me that it might actually do the trick to some degree, though I'm afraid this is still not what we are looking for. While Mistral is asynchronous and event-driven, this particular design is not and would still force us to store the engine in memory and therefore limit our means of scalability. The lazy engine (or better controller) I have proposed is asynchronous at its core and would fit the needs for both of us (since it's much easier to make sync from async, rather than backwards). Regarding the retries, while it might work with the current flow design, I doubt it would work with conditional transitions. The attempt to build a repeater by incapsulating the tasks into sub-flow will basically means that every transition they produce will be in that flow and you can't leave it until they are all finished. The whole idea of sub-flows within the scope of direct conditional transitions is a bit unclear to me (and probably us all) at the moment, though I'm trying to rely on them only as a means to lesser the complexity. [1] https://review.openstack.org/#/c/86470 -- Kirill Izotov пятница, 11 апреля 2014 г. в 23:47, Joshua Harlow написал: Thanks for the write-up krill. Also some adjustments, Both points are good, and putting some of this on @ https://etherpad.openstack.org/p/taskflow-mistral-details so that we can have it actively noted (feel free to adjust it). I think ivan is working on some docs/code/… for the lazy engine idea, so hopefully we can get back soon with that. Lets see what comes out of that effort and iterate on that. For (2), our are mostly correct about unconditional execution although [1] does now change this, and there are a few active reviews that are being worked [3] on to fit this mistral use-case better. I believe [2] can help move in this direction, ivans ideas I think will also push it a little farther to. Of course lets work together to make sure they fit the best so that taskflow mistral openstack can be the best it can be (pigeons not included). Can we also make sure the small issues are noted somewhere (maybe in the above etherpad??). Thanks! [1] https://wiki.openstack.org/wiki/TaskFlow#Retries [2] https://review.openstack.org/#/c/86470 [3] https://review.openstack.org/#/q/status:open+project:openstack/taskflow,n,z From: Kirill Izotov enyk...@stackstorm.com (mailto:enyk...@stackstorm.com) Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org (mailto:openstack-dev@lists.openstack.org) Date: Thursday, April 10, 2014 at 9:20 PM To: OpenStack-dev@lists.openstack.org (mailto:OpenStack-dev@lists.openstack.org) OpenStack-dev@lists.openstack.org (mailto:OpenStack-dev@lists.openstack.org) Subject: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary Hi everyone, This is a summary to the prototype integration we did not too long ago: http://github.com/enykeev/mistral/pull/1. Hope it would shed some light on the aspects of the integration we are struggling with. There is a possibility to build Mistral on top of TaskFlow as a library, but in order to meet the requirements dictated by Mistral users and use cases, both Mistral and TaskFlow should change. There are two main sides of the story
Re: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary
They are all parts of conditional transitions: every task should have a number of possible transitions; each transition consist of a reference to the task we want to transit to and the condition that should evaluate to true for transition to start. At that point, I'd say that it perfectly fine for TaskFlow to evaluate python conditions rather than implementing YAQL, though there should be a place for us to pass the condition evaluation logic we are using. -- Kirill Izotov вторник, 15 апреля 2014 г. в 8:02, Joshua Harlow написал: Can we describe exactly what references, direct transition, expression evaluation are doing in #2. Expression evaluation especially seems to be an odd one, what's wrong with pythons expression evaluation? I can't quite see why that would/should exist in taskflow. I can see it being implemented in mistral, where mistral converts whatever DSL it wants into taskflow primitives and then taskflow runs the code; this decoupling ensures that taskflow does not force a DSL on people that want to use taskflow as a python library (this kind of restriction imho isn't acceptable for a library to do, and limits taskflows own usage and integration). Thanks, Josh From: Dmitri Zimine d...@stackstorm.com (mailto:d...@stackstorm.com) Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org (mailto:openstack-dev@lists.openstack.org) Date: Friday, April 11, 2014 at 9:55 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org (mailto:openstack-dev@lists.openstack.org) Subject: [openstack-dev] [Mistral][TaskFlow] Mistral-TaskFlow Summary We prototyped Mistral / TaskFlow integration and have a follow-up discussions. SUMMARY: Mistral (Workflow Service) can embed TaskFlow as a workflow library, with some required modifications to function resliently as a service, and for smooth integration. However, the TaskFlow flow controls are insufficient for Mistral use cases. Details discussed on other thirds. The prototype scope - [0 (https://etherpad.openstack.org/p/mistral-taskflow-prototype)]; code and discussion - [1 (https://github.com/enykeev/mistral/pull/1)] and techical highlights - [2 (http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html)]. DETAILS: 1) Embedding TaskFlow inside Mistral: * Required: make the engine lazy [3 (http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html)], [4 (http://paste.openstack.org/show/75389/)].This is required to support long-running delegates and not loose tasks when engine manager process restarts. * Persistence: need clarity how to replace or mix-in TaskFlow persistence with Mistral persistence. Renat is taking a look. * Declaring Flows in YAML DSL: done for simplest flow. Need to prototype for data flow. Rich flow controls are missing in TaskFlow for a representative prototype. * ActionRunners vs Taskflow Workers - not prototyped. Not a risk: both Mistral and TaskFlow implementations work. But we shall resolve the overlap. * Ignored for now - unlikely any risks: Keystone integration, Mistral event scheduler, Mistral declarative services and action definition. 2) TaskFlow library features * Must: flow control - conditional transitions, references, expression evaluation, to express real-life workflows [5 (https://github.com/dzimine/mistral-workflows/tree/add-usecases)]. The required flow control primitives are 1) repeater 2) flow in flow 3) direct transition 4) conditional transition 5) multiple data. TaskFlow has 1) and 2), need to add 3/4/5. * Other details and smaller requests are in the discussion [1 (https://github.com/enykeev/mistral/pull/1)] 3) Next Steps proposed: * Mistal team: summarize the requirements discussed and agreed on [2 (http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html)] and [3 (http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html)] * Mistral team: code sample (tests?) on how Mistral would like to consume TaskFlow lazy engine * Taskflow team: Provide a design for alternative TaskExecutor approach (prototypes, boxes, arrows, crayons :)) * Decide on lazy engine * Move the discussion on other elements on integration. References: [0] The scope of the prototype: https://etherpad.openstack.org/p/mistral-taskflow-prototype [1] Prototype code and discussion https://github.com/enykeev/mistral/pull/1 [2] Techical summary http://lists.openstack.org/pipermail/openstack-dev/2014-April/032461.html [3] Email discussion on TaskFlow lazy eninge http://lists.openstack.org/pipermail/openstack-dev/2014-March/031134.html [4] IRC discussion Mistral/Taskflow http://paste.openstack.org/show/75389/ [5
Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary
Thank for pointing that out, Joshua. I had a look on [1] and it seems to me that it might actually do the trick to some degree, though I'm afraid this is still not what we are looking for. While Mistral is asynchronous and event-driven, this particular design is not and would still force us to store the engine in memory and therefore limit our means of scalability. The lazy engine (or better controller) I have proposed is asynchronous at its core and would fit the needs for both of us (since it's much easier to make sync from async, rather than backwards). Regarding the retries, while it might work with the current flow design, I doubt it would work with conditional transitions. The attempt to build a repeater by incapsulating the tasks into sub-flow will basically means that every transition they produce will be in that flow and you can't leave it until they are all finished. The whole idea of sub-flows within the scope of direct conditional transitions is a bit unclear to me (and probably us all) at the moment, though I'm trying to rely on them only as a means to lesser the complexity. [1] https://review.openstack.org/#/c/86470 -- Kirill Izotov пятница, 11 апреля 2014 г. в 23:47, Joshua Harlow написал: Thanks for the write-up krill. Also some adjustments, Both points are good, and putting some of this on @ https://etherpad.openstack.org/p/taskflow-mistral-details so that we can have it actively noted (feel free to adjust it). I think ivan is working on some docs/code/… for the lazy engine idea, so hopefully we can get back soon with that. Lets see what comes out of that effort and iterate on that. For (2), our are mostly correct about unconditional execution although [1] does now change this, and there are a few active reviews that are being worked [3] on to fit this mistral use-case better. I believe [2] can help move in this direction, ivans ideas I think will also push it a little farther to. Of course lets work together to make sure they fit the best so that taskflow mistral openstack can be the best it can be (pigeons not included). Can we also make sure the small issues are noted somewhere (maybe in the above etherpad??). Thanks! [1] https://wiki.openstack.org/wiki/TaskFlow#Retries [2] https://review.openstack.org/#/c/86470 [3] https://review.openstack.org/#/q/status:open+project:openstack/taskflow,n,z From: Kirill Izotov enyk...@stackstorm.com (mailto:enyk...@stackstorm.com) Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org (mailto:openstack-dev@lists.openstack.org) Date: Thursday, April 10, 2014 at 9:20 PM To: OpenStack-dev@lists.openstack.org (mailto:OpenStack-dev@lists.openstack.org) OpenStack-dev@lists.openstack.org (mailto:OpenStack-dev@lists.openstack.org) Subject: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary Hi everyone, This is a summary to the prototype integration we did not too long ago: http://github.com/enykeev/mistral/pull/1. Hope it would shed some light on the aspects of the integration we are struggling with. There is a possibility to build Mistral on top of TaskFlow as a library, but in order to meet the requirements dictated by Mistral users and use cases, both Mistral and TaskFlow should change. There are two main sides of the story. One is engine. The other is flow control capabilities. 1) THE ENGINE The current TaskFlow implementation of engine doesn't fit Mistral needs because it is synchronous, it blocks the thread, it requires us to store the reference to the particular engine to be able to get its status and suspend the execution and it lacks long-running task compatibility. To fix this problem in a solid and maintainable way, we need to split the engine into its synchronous and asynchronous counterparts. Lazy engine should be async and atomic, it should not have its own state, instead it should rely on some kind of global state (db or in-memory, depending on a type of application). It should have at least two methods: run and task_complete. Run method should calculate the first batch of tasks and schedule them for executing (either put them in queue or spawn the threads). Task_complete should mark a certain task to be completed and then schedule the next batch of tasks that became available due to resolution of this one. The desired use of lazy engine in Mistral is illustrated here: https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign#Big_Picture. It should support long running tasks and survive engine process restart without loosing the state of the running actions. So it must be passive (lazy) and persistent. On Mistral side we are using Lazy engine by patching async.run directly to the API (or engine queue) and async.task_complete
[openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary
Hi everyone, This is a summary to the prototype integration we did not too long ago: http://github.com/enykeev/mistral/pull/1. Hope it would shed some light on the aspects of the integration we are struggling with. There is a possibility to build Mistral on top of TaskFlow as a library, but in order to meet the requirements dictated by Mistral users and use cases, both Mistral and TaskFlow should change. There are two main sides of the story. One is engine. The other is flow control capabilities. 1) THE ENGINE The current TaskFlow implementation of engine doesn't fit Mistral needs because it is synchronous, it blocks the thread, it requires us to store the reference to the particular engine to be able to get its status and suspend the execution and it lacks long-running task compatibility. To fix this problem in a solid and maintainable way, we need to split the engine into its synchronous and asynchronous counterparts. Lazy engine should be async and atomic, it should not have its own state, instead it should rely on some kind of global state (db or in-memory, depending on a type of application). It should have at least two methods: run and task_complete. Run method should calculate the first batch of tasks and schedule them for executing (either put them in queue or spawn the threads). Task_complete should mark a certain task to be completed and then schedule the next batch of tasks that became available due to resolution of this one. The desired use of lazy engine in Mistral is illustrated here: https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign#Big_Picture. It should support long running tasks and survive engine process restart without loosing the state of the running actions. So it must be passive (lazy) and persistent. On Mistral side we are using Lazy engine by patching async.run directly to the API (or engine queue) and async.task_complete to the worker queue result channel (and the API for long running tasks). We are still sharing the same graph_analyzer, but instead of relying on loop and Futures, we are handling the execution ourselves in a scalable and robust way. Then, on top of it you can build a sync engine by introducing Futures. You are using async.run() to schedule tasks by transforming them to Futures and then starting a loop, checking Futures for completion and sending their results to async.task_complete() which would produce even more Futures to check over. Just the same way TaskFlow do it right now. The reason I'm proposing to extract Futures from async engine is because they won't work if we have multiple engine processes that should handle the task results concurrently (and without that there will be no scalability). 2) THE FLOW CONTROL CAPABILITIES Since we treat TaskFlow as a library we expect them to provide us with a number of primitives to build our workflow with them. Most important of them to us for the moment are Direct Transitions, and Conditional Transitions. The current implementation of flow transitions in TaskFlow are built on top of data flow dependencies where each task provides some data to the flow and requires some data to be present prior being executed. In other words, you are starting to build your flow tree from the last task through the first one by adding their requirements to the tree. All the tasks of successfully finished flow should be successfully finished too. If one of the tasks finishes with error, the whole flow will be reverted back to its initial state unconditionally. At the same time, Mistral use cases require direct control on the order of the task execution, with top-to-bottom scheme where the next task will be determined based on the results of the execution of the current one. This way to successfully finish a flow you don't have to execute all tasks in it. Besides, the error in execution of a particular task may cause execution of another one. The workflow examples (in pseudo DSL) are here: https://github.com/dzimine/mistral-workflows/tree/add-usecases There is also a handful of small issues, but these two differences cover most basic parts of TaskFlow thus block us from integration and require substantial changes in TaskFlow engine design. Inability to make such changes will directly result in Mistral not being able to meet its requirements and thus rendering the whole project useless for its users. -- Kirill Izotov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] How Mistral handling long running delegate tasks
Then, we can make task executor interface public and allow clients to provide their own task executors. It will be possible then for Mistral to implement its own task executor, or several, and share the executors between all the engine instances. I'm afraid that if we start to tear apart the TaskFlow engine, it would quickly become a mess to support. Besides, the amount of things left to integrate after we throw out engine might be so low it proof the whole process of integration to be just nominal and we are back to square one. Any way, task execution is the part that least bothers me, both graph action and the engine itself is where the pain will be. That is part of our public API, it is stable and good enough. Basically, I don't think this API needs any major change. But whatever should and will be done about it, I daresay all that work can be done without affecting API more then I described above. I completely agree that we should not change the public API of the sync engine, especially the one in helpers. What we need is, on the contrary, a low level construct that would do the number of things i stated previously, but will be a part of public API of TaskFlow so we can be sure it would work exactly the same way it worked yesterday. -- Kirill Izotov пятница, 4 апреля 2014 г. в 2:04, Ivan Melnikov написал: I'm trying to catch up this rather long and interesting discussion, sorry for somewhat late reply. I can see aspects of 'lazy model' support in TaskFlow: - how tasks are executed and reverted - how flows are run - how engine works internally Let me address those aspects separately. == Executing and reverting tasks == I think that should be done via different interface then running a flow (or scheduling it to run), as it is completely different thing. In current TaskFlow this interface is called task executor: https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/executor.py#L57 That is actually how our WorkerBasedEngine was implemented: it's the same engine with special task executor that schedules tasks on worker instead of running task code locally. Task executors are not aware of flows by design, all they do is executing and reverting tasks. That means that task executors can be easily shared between engines if that's wanted. Current TaskExecutorBase interface uses futures (PEP 3148-like). When I proposed it, futures looked like good tool for the task at hand (see e.g. async task etherpad https://etherpad.openstack.org/p/async-taskflow-tasks) Now it may be time to reconsider that: having one future object per running task may become a scalability issue. It may be worth to use callbacks instead. It should not be too hard to refactor current engine for that. Also, as TaskExecutorBase is an internal API, there should not be any compatibility issues. Then, we can make task executor interface public and allow clients to provide their own task executors. It will be possible then for Mistral to implement its own task executor, or several, and share the executors between all the engine instances. You can call it a plan;) == Running flows == To run the flow TaskFlow client uses engine interface; also, there are few of helper functions provided for convenience: http://docs.openstack.org/developer/taskflow/engines.html#module-taskflow.engines.base http://docs.openstack.org/developer/taskflow/engines.html#creating-engines That is part of our public API, it is stable and good enough. Basically, I don't think this API needs any major change. Maybe it worth to add function or method to schedule running flow without actually waiting for flow completion (at least, it was on my top secret TODO list for quite a long time). == Engine internals == Each engine eats resources, like thread it runs on; using these resources to run one flow only is somewhat wasteful. Some work is already planned to address this situation (see e.g. https://blueprints.launchpad.net/taskflow/+spec/share-engine-thread). Also, it might be good idea to implement different 'type' of engine to support 'lazy' model, as Joshua suggests. But whatever should and will be done about it, I daresay all that work can be done without affecting API more then I described above. -- WBR, Ivan A. Melnikov ... tasks must flow ... On 02.04.2014 01:51, Dmitri Zimine wrote: Even more responses inline :) [...] ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org (mailto:OpenStack-dev@lists.openstack.org) http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Engine overview and proposal
I have an idea regarding engine design i want to share with you. But first, it seems like we need a small overview of the current implementations. I'm not sure how ML will react on a bunch of ASCII graphs, so here is an etherpad: https://etherpad.openstack.org/p/mistral-engine-overview-and-proposal What do you think, guys, is this the way to go? -- Kirill Izotov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [openstack][Mistral] Adding new core reviewers
Sure, +1 for both. -- Kirill Izotov четверг, 20 марта 2014 г. в 3:52, Dmitri Zimine написал: +1 for Nikolay - he's indeed the most involved in all aspect. and yes I am up to doing it. DZ. On Mar 19, 2014, at 4:35 AM, Renat Akhmerov rakhme...@mirantis.com (mailto:rakhme...@mirantis.com) wrote: Team, So far I’ve been just the only one core member of the team. I started feeling lonely :) Since the project team and the project itself has now grown (thanks to StackStorm and Intel) I think it’s time to think about extending the core team. I would propose: Nikolay Makhotkin (nmakhotkin at launchpad). He's been working on the project since almost the very beginning and made significant contribution (design, reviews, code). Dmitri Zimine (i-dz at launchpad). Dmitri joined the project about 2 months ago. Since then he’s made a series of important high-quality commits, a lot of valuable reviews and, IMO most importantly, he has a solid vision of the project in general (requirements, use cases, comparison to other technologies) and has a pro-active viewpoint in all our discussions. Thoughts? Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org (mailto:OpenStack-dev@lists.openstack.org) http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org (mailto:OpenStack-dev@lists.openstack.org) http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
