Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary
Ivan, any update/progress on this? Thanks, DZ. On Apr 15, 2014, at 10:21 AM, Joshua Harlow harlo...@yahoo-inc.com wrote: Well Ivan afaik is thinking through it, but being a community project its not exactly easy to put estimations or timelines on things (I don't control Ivan, or others). Likely faster if u guys want to get involved. -Josh From: Renat Akhmerov rakhme...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Tuesday, April 15, 2014 at 12:35 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary On 15 Apr 2014, at 11:13, Joshua Harlow harlo...@yahoo-inc.com wrote: Sure, its not the fully complete lazy_engine, but piece by piece we can get there. Did you make any estimations when it could happen? :) Of course code/contributions are welcome, as such things will benefit more than just mistral, but openstack as a whole :-) OK. From: Kirill Izotov enyk...@stackstorm.com The whole idea of sub-flows within the scope of direct conditional transitions is a bit unclear to me (and probably us all) at the moment, though I'm trying to rely on them only as a means to lesser the complexity. Yes, eventually it’s for reducing complexity. I would just add that it opens wide range of opportunities like: ability to combine multiple physically independent workflows reusability (using one workflow as a part of another) isolation (different namespaces for data flow contexts etc) Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary
Since you have mentioned that, I'm kind of interested, who else is going to benefit from this transition? I'm asking because everyone tells me that it's essential for us to work together, though my senses tells me that we are starting from different prerequisites, targeting different use cases and taking the entirely different approaches to solve the problem. Actually, I'd say we have more differences than the things we have in common. Did you hear the same requests from anyone of your current users? May be we should invite them for discussion, just to make sure we would not have to redo it again. -- Kirill Izotov вторник, 15 апреля 2014 г. в 11:13, Joshua Harlow написал: Sure, its not the fully complete lazy_engine, but piece by piece we can get there. Of course code/contributions are welcome, as such things will benefit more than just mistral, but openstack as a whole :-) -Josh From: Kirill Izotov enyk...@stackstorm.com (mailto:enyk...@stackstorm.com) Date: Monday, April 14, 2014 at 9:02 PM To: Joshua Harlow harlo...@yahoo-inc.com (mailto:harlo...@yahoo-inc.com) Cc: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org (mailto:openstack-dev@lists.openstack.org) Subject: Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary Thank for pointing that out, Joshua. I had a look on [1] and it seems to me that it might actually do the trick to some degree, though I'm afraid this is still not what we are looking for. While Mistral is asynchronous and event-driven, this particular design is not and would still force us to store the engine in memory and therefore limit our means of scalability. The lazy engine (or better controller) I have proposed is asynchronous at its core and would fit the needs for both of us (since it's much easier to make sync from async, rather than backwards). Regarding the retries, while it might work with the current flow design, I doubt it would work with conditional transitions. The attempt to build a repeater by incapsulating the tasks into sub-flow will basically means that every transition they produce will be in that flow and you can't leave it until they are all finished. The whole idea of sub-flows within the scope of direct conditional transitions is a bit unclear to me (and probably us all) at the moment, though I'm trying to rely on them only as a means to lesser the complexity. [1] https://review.openstack.org/#/c/86470 -- Kirill Izotov пятница, 11 апреля 2014 г. в 23:47, Joshua Harlow написал: Thanks for the write-up krill. Also some adjustments, Both points are good, and putting some of this on @ https://etherpad.openstack.org/p/taskflow-mistral-details so that we can have it actively noted (feel free to adjust it). I think ivan is working on some docs/code/… for the lazy engine idea, so hopefully we can get back soon with that. Lets see what comes out of that effort and iterate on that. For (2), our are mostly correct about unconditional execution although [1] does now change this, and there are a few active reviews that are being worked [3] on to fit this mistral use-case better. I believe [2] can help move in this direction, ivans ideas I think will also push it a little farther to. Of course lets work together to make sure they fit the best so that taskflow mistral openstack can be the best it can be (pigeons not included). Can we also make sure the small issues are noted somewhere (maybe in the above etherpad??). Thanks! [1] https://wiki.openstack.org/wiki/TaskFlow#Retries [2] https://review.openstack.org/#/c/86470 [3] https://review.openstack.org/#/q/status:open+project:openstack/taskflow,n,z From: Kirill Izotov enyk...@stackstorm.com (mailto:enyk...@stackstorm.com) Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org (mailto:openstack-dev@lists.openstack.org) Date: Thursday, April 10, 2014 at 9:20 PM To: OpenStack-dev@lists.openstack.org (mailto:OpenStack-dev@lists.openstack.org) OpenStack-dev@lists.openstack.org (mailto:OpenStack-dev@lists.openstack.org) Subject: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary Hi everyone, This is a summary to the prototype integration we did not too long ago: http://github.com/enykeev/mistral/pull/1. Hope it would shed some light on the aspects of the integration we are struggling with. There is a possibility to build Mistral on top of TaskFlow as a library, but in order to meet the requirements dictated by Mistral users and use cases, both Mistral and TaskFlow should change. There are two main sides of the story
Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary
On 15 Apr 2014, at 11:13, Joshua Harlow harlo...@yahoo-inc.com wrote: Sure, its not the fully complete lazy_engine, but piece by piece we can get there. Did you make any estimations when it could happen? :) Of course code/contributions are welcome, as such things will benefit more than just mistral, but openstack as a whole :-) OK. From: Kirill Izotov enyk...@stackstorm.com The whole idea of sub-flows within the scope of direct conditional transitions is a bit unclear to me (and probably us all) at the moment, though I'm trying to rely on them only as a means to lesser the complexity. Yes, eventually it’s for reducing complexity. I would just add that it opens wide range of opportunities like: ability to combine multiple physically independent workflows reusability (using one workflow as a part of another) isolation (different namespaces for data flow contexts etc) Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary
Well Ivan afaik is thinking through it, but being a community project its not exactly easy to put estimations or timelines on things (I don't control Ivan, or others). Likely faster if u guys want to get involved. -Josh From: Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Tuesday, April 15, 2014 at 12:35 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary On 15 Apr 2014, at 11:13, Joshua Harlow harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com wrote: Sure, its not the fully complete lazy_engine, but piece by piece we can get there. Did you make any estimations when it could happen? :) Of course code/contributions are welcome, as such things will benefit more than just mistral, but openstack as a whole :-) OK. From: Kirill Izotov enyk...@stackstorm.commailto:enyk...@stackstorm.com The whole idea of sub-flows within the scope of direct conditional transitions is a bit unclear to me (and probably us all) at the moment, though I'm trying to rely on them only as a means to lesser the complexity. Yes, eventually it’s for reducing complexity. I would just add that it opens wide range of opportunities like: * ability to combine multiple physically independent workflows * reusability (using one workflow as a part of another) * isolation (different namespaces for data flow contexts etc) Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary
Thank for pointing that out, Joshua. I had a look on [1] and it seems to me that it might actually do the trick to some degree, though I'm afraid this is still not what we are looking for. While Mistral is asynchronous and event-driven, this particular design is not and would still force us to store the engine in memory and therefore limit our means of scalability. The lazy engine (or better controller) I have proposed is asynchronous at its core and would fit the needs for both of us (since it's much easier to make sync from async, rather than backwards). Regarding the retries, while it might work with the current flow design, I doubt it would work with conditional transitions. The attempt to build a repeater by incapsulating the tasks into sub-flow will basically means that every transition they produce will be in that flow and you can't leave it until they are all finished. The whole idea of sub-flows within the scope of direct conditional transitions is a bit unclear to me (and probably us all) at the moment, though I'm trying to rely on them only as a means to lesser the complexity. [1] https://review.openstack.org/#/c/86470 -- Kirill Izotov пятница, 11 апреля 2014 г. в 23:47, Joshua Harlow написал: Thanks for the write-up krill. Also some adjustments, Both points are good, and putting some of this on @ https://etherpad.openstack.org/p/taskflow-mistral-details so that we can have it actively noted (feel free to adjust it). I think ivan is working on some docs/code/… for the lazy engine idea, so hopefully we can get back soon with that. Lets see what comes out of that effort and iterate on that. For (2), our are mostly correct about unconditional execution although [1] does now change this, and there are a few active reviews that are being worked [3] on to fit this mistral use-case better. I believe [2] can help move in this direction, ivans ideas I think will also push it a little farther to. Of course lets work together to make sure they fit the best so that taskflow mistral openstack can be the best it can be (pigeons not included). Can we also make sure the small issues are noted somewhere (maybe in the above etherpad??). Thanks! [1] https://wiki.openstack.org/wiki/TaskFlow#Retries [2] https://review.openstack.org/#/c/86470 [3] https://review.openstack.org/#/q/status:open+project:openstack/taskflow,n,z From: Kirill Izotov enyk...@stackstorm.com (mailto:enyk...@stackstorm.com) Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org (mailto:openstack-dev@lists.openstack.org) Date: Thursday, April 10, 2014 at 9:20 PM To: OpenStack-dev@lists.openstack.org (mailto:OpenStack-dev@lists.openstack.org) OpenStack-dev@lists.openstack.org (mailto:OpenStack-dev@lists.openstack.org) Subject: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary Hi everyone, This is a summary to the prototype integration we did not too long ago: http://github.com/enykeev/mistral/pull/1. Hope it would shed some light on the aspects of the integration we are struggling with. There is a possibility to build Mistral on top of TaskFlow as a library, but in order to meet the requirements dictated by Mistral users and use cases, both Mistral and TaskFlow should change. There are two main sides of the story. One is engine. The other is flow control capabilities. 1) THE ENGINE The current TaskFlow implementation of engine doesn't fit Mistral needs because it is synchronous, it blocks the thread, it requires us to store the reference to the particular engine to be able to get its status and suspend the execution and it lacks long-running task compatibility. To fix this problem in a solid and maintainable way, we need to split the engine into its synchronous and asynchronous counterparts. Lazy engine should be async and atomic, it should not have its own state, instead it should rely on some kind of global state (db or in-memory, depending on a type of application). It should have at least two methods: run and task_complete. Run method should calculate the first batch of tasks and schedule them for executing (either put them in queue or spawn the threads). Task_complete should mark a certain task to be completed and then schedule the next batch of tasks that became available due to resolution of this one. The desired use of lazy engine in Mistral is illustrated here: https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign#Big_Picture. It should support long running tasks and survive engine process restart without loosing the state of the running actions. So it must be passive (lazy) and persistent. On Mistral side we are using Lazy engine by patching async.run directly to the API (or engine queue) and async.task_complete to the
Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary
Sure, its not the fully complete lazy_engine, but piece by piece we can get there. Of course code/contributions are welcome, as such things will benefit more than just mistral, but openstack as a whole :-) -Josh From: Kirill Izotov enyk...@stackstorm.commailto:enyk...@stackstorm.com Date: Monday, April 14, 2014 at 9:02 PM To: Joshua Harlow harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com Cc: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary Thank for pointing that out, Joshua. I had a look on [1] and it seems to me that it might actually do the trick to some degree, though I'm afraid this is still not what we are looking for. While Mistral is asynchronous and event-driven, this particular design is not and would still force us to store the engine in memory and therefore limit our means of scalability. The lazy engine (or better controller) I have proposed is asynchronous at its core and would fit the needs for both of us (since it's much easier to make sync from async, rather than backwards). Regarding the retries, while it might work with the current flow design, I doubt it would work with conditional transitions. The attempt to build a repeater by incapsulating the tasks into sub-flow will basically means that every transition they produce will be in that flow and you can't leave it until they are all finished. The whole idea of sub-flows within the scope of direct conditional transitions is a bit unclear to me (and probably us all) at the moment, though I'm trying to rely on them only as a means to lesser the complexity. [1] https://review.openstack.org/#/c/86470 -- Kirill Izotov пятница, 11 апреля 2014 г. в 23:47, Joshua Harlow написал: Thanks for the write-up krill. Also some adjustments, Both points are good, and putting some of this on @ https://etherpad.openstack.org/p/taskflow-mistral-details so that we can have it actively noted (feel free to adjust it). I think ivan is working on some docs/code/… for the lazy engine idea, so hopefully we can get back soon with that. Lets see what comes out of that effort and iterate on that. For (2), our are mostly correct about unconditional execution although [1] does now change this, and there are a few active reviews that are being worked [3] on to fit this mistral use-case better. I believe [2] can help move in this direction, ivans ideas I think will also push it a little farther to. Of course lets work together to make sure they fit the best so that taskflow mistral openstack can be the best it can be (pigeons not included). Can we also make sure the small issues are noted somewhere (maybe in the above etherpad??). Thanks! [1] https://wiki.openstack.org/wiki/TaskFlow#Retries [2] https://review.openstack.org/#/c/86470 [3] https://review.openstack.org/#/q/status:open+project:openstack/taskflow,n,z From: Kirill Izotov enyk...@stackstorm.commailto:enyk...@stackstorm.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Thursday, April 10, 2014 at 9:20 PM To: OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org Subject: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary Hi everyone, This is a summary to the prototype integration we did not too long ago: http://github.com/enykeev/mistral/pull/1. Hope it would shed some light on the aspects of the integration we are struggling with. There is a possibility to build Mistral on top of TaskFlow as a library, but in order to meet the requirements dictated by Mistral users and use cases, both Mistral and TaskFlow should change. There are two main sides of the story. One is engine. The other is flow control capabilities. 1) THE ENGINE The current TaskFlow implementation of engine doesn't fit Mistral needs because it is synchronous, it blocks the thread, it requires us to store the reference to the particular engine to be able to get its status and suspend the execution and it lacks long-running task compatibility. To fix this problem in a solid and maintainable way, we need to split the engine into its synchronous and asynchronous counterparts. Lazy engine should be async and atomic, it should not have its own state, instead it should rely on some kind of global state (db or in-memory, depending on a type of application). It should have at least two methods: run and task_complete. Run method should calculate the first batch of tasks and schedule them for executing (either put them in queue or spawn the threads). Task_complete should mark a certain task to be completed and then schedule the next batch of tasks that became available due to resolution of this one
Re: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary
Thanks for the write-up krill. Also some adjustments, Both points are good, and putting some of this on @ https://etherpad.openstack.org/p/taskflow-mistral-details so that we can have it actively noted (feel free to adjust it). I think ivan is working on some docs/code/… for the lazy engine idea, so hopefully we can get back soon with that. Lets see what comes out of that effort and iterate on that. For (2), our are mostly correct about unconditional execution although [1] does now change this, and there are a few active reviews that are being worked [3] on to fit this mistral use-case better. I believe [2] can help move in this direction, ivans ideas I think will also push it a little farther to. Of course lets work together to make sure they fit the best so that taskflow mistral openstack can be the best it can be (pigeons not included). Can we also make sure the small issues are noted somewhere (maybe in the above etherpad??). Thanks! [1] https://wiki.openstack.org/wiki/TaskFlow#Retries [2] https://review.openstack.org/#/c/86470 [3] https://review.openstack.org/#/q/status:open+project:openstack/taskflow,n,z From: Kirill Izotov enyk...@stackstorm.commailto:enyk...@stackstorm.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Thursday, April 10, 2014 at 9:20 PM To: OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org Subject: [openstack-dev] [mistral] [taskflow] Mistral TaskFlow integration summary Hi everyone, This is a summary to the prototype integration we did not too long ago: http://github.com/enykeev/mistral/pull/1. Hope it would shed some light on the aspects of the integration we are struggling with. There is a possibility to build Mistral on top of TaskFlow as a library, but in order to meet the requirements dictated by Mistral users and use cases, both Mistral and TaskFlow should change. There are two main sides of the story. One is engine. The other is flow control capabilities. 1) THE ENGINE The current TaskFlow implementation of engine doesn't fit Mistral needs because it is synchronous, it blocks the thread, it requires us to store the reference to the particular engine to be able to get its status and suspend the execution and it lacks long-running task compatibility. To fix this problem in a solid and maintainable way, we need to split the engine into its synchronous and asynchronous counterparts. Lazy engine should be async and atomic, it should not have its own state, instead it should rely on some kind of global state (db or in-memory, depending on a type of application). It should have at least two methods: run and task_complete. Run method should calculate the first batch of tasks and schedule them for executing (either put them in queue or spawn the threads). Task_complete should mark a certain task to be completed and then schedule the next batch of tasks that became available due to resolution of this one. The desired use of lazy engine in Mistral is illustrated here: https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign#Big_Picture. It should support long running tasks and survive engine process restart without loosing the state of the running actions. So it must be passive (lazy) and persistent. On Mistral side we are using Lazy engine by patching async.run directly to the API (or engine queue) and async.task_complete to the worker queue result channel (and the API for long running tasks). We are still sharing the same graph_analyzer, but instead of relying on loop and Futures, we are handling the execution ourselves in a scalable and robust way. Then, on top of it you can build a sync engine by introducing Futures. You are using async.run() to schedule tasks by transforming them to Futures and then starting a loop, checking Futures for completion and sending their results to async.task_complete() which would produce even more Futures to check over. Just the same way TaskFlow do it right now. The reason I'm proposing to extract Futures from async engine is because they won't work if we have multiple engine processes that should handle the task results concurrently (and without that there will be no scalability). 2) THE FLOW CONTROL CAPABILITIES Since we treat TaskFlow as a library we expect them to provide us with a number of primitives to build our workflow with them. Most important of them to us for the moment are Direct Transitions, and Conditional Transitions. The current implementation of flow transitions in TaskFlow are built on top of data flow dependencies where each task provides some data to the flow and requires some data to be present prior being executed. In other words, you are starting to build your flow tree from the last task through the first one by adding their requirements to the
