Re: [openstack-dev] [Mistral] Actions design BP
On 19 Mar 2014, at 02:08, Joshua Harlow wrote: > is mistral planning doing this, throwing way the POC and rewriting it as a > non-POC using the ideas learned from the POC?)[http://tinyurl.com/lbz293s]. Yes, if needed. I don’t think it’s too important and I agree that terminology here may be a thing that makes some confusion. If we decide most of the parts are fine to be just evolving (instead of rewriting them) we can use them as their are. > For the 'asynchronous manner' discussion see http://tinyurl.com/n3v9lt8; I'm > still not sure why u would want to make is_sync/is_async a primitive concept > in a workflow system, shouldn't this be only up to the entity running the > workflow to decide? > Why is a task allowed to be sync/async I would ask the opposite: “Why a task should always be sync?” Where does this limitation come from? Any long lasting and/or resource intensive task should be logically considered async. Async here may not be a good word even. I apologize if it’s a problem for understanding. The main thing here is that a workflow system should be able not to hold local state to keep track of these heavy tasks. Otherwise, the consequences are well-known: hard to scale, hard to make it durable, etc etc. Some of the tasks may last weeks or even months (real life examples from the customers). If the engine is used as a hosted service then resource starvation may easily become an absolutely problem. From my experience, assumptions like that made at design phase used to make production systems go down. In other words, all the flowering variety of side-effects of Distributed Programming come from that. So we decided to put this conceptual understanding into a core of the system. Joshua, can we start working on that in TaskFlow? We’re ready to help and contribute (myself and other folks from the team). > , that has major side-effects for state-persistence, resumption (and to me is > a incorrect abstraction to provide) and general workflow execution control, > I'd be very careful with this (which is why I am hesitant to add it without > much much more discussion). Of course, it has major effects. It’s what we’ve been emphasising as a fundamental difference. In that regard, one of the interesting ideas is to thing about the role that TaskFlow in OpenStack community. Should it be dealing with, say, with persistence not being a service and not being able to fully address HA? I know your point of view on that (that at least it could provide facilities to address HA partially etc), but that’s a topic to think about. I’d suggest we start discussing that too. >> So we actually talked to people a lot (including Josh) and provided this >> reasoning when this question raised again. Reaction was nearly always >> positive and made a lot of sense to customers and developers. >> >> Thought #2: A library shouldn't drive a project where it’s used. > > > To me this assumes said library is fixed in stone, can't be changed, and > can't be evolved. If a library is 'dead/vaporware' then sure, I would 100% > agree with this, but all libraries in openstack do not fit into the later > category; and those libraries can be evolved/developed/improved. As a > community I think it is our goal to grow the libraries in the community (not > reduce them or avoid them, as this is not benefical for the community). I > think doug put this very well his essay @ http://tinyurl.com/lr9wvfl and imho > we need to embrace evolving libraries in all projects, not avoiding them due > to thoughts like this. 100% agree that OS libraries are not fixed in stone, no questions. I meant a totally different thing: the process of evolution. TaskFlow is a beautiful well-written library, at least I think so. But we shouldn’t build a project around a library in the first place. We should be building a project and generating requirements to a library. If the library is ready to change it's wonderful. If it is not, I’m totally against of adjusting the project feature set so that it can be implemented using the library capabilities. Am I missing something? For example, when I had a project where we used a persistence framework and it was ok for a while but at some point we realized that we needed lazy-loading that was missing in that framework and since the team of that framework said “it’s against our principles, you better change your vision of the project” we said ok, threw it away and started using Hibernate and eventually all became happy including the customer. As simple as that. And it makes a lot of sense to me. “Use whatever works well for you”. At the same time, I clearly understand that in OpenStack we can’t do like that. And honestly, we don’t want, since eventually we’re working on the same huge thing: cloud. Well, actually we can ignore others but not for a long time :) Rule is a rule, TC folks honestly say it was done intentionally to make people collaborate, otherwise we wo
Re: [openstack-dev] [Mistral] Actions design BP
From: Renat Akhmerov mailto:rakhme...@mirantis.com>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Date: Monday, March 17, 2014 at 10:51 PM To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack-dev] [Mistral] Actions design BP On 18 Mar 2014, at 01:32, Joshua Harlow mailto:harlo...@yahoo-inc.com>> wrote: To further this lets continue working on https://etherpad.openstack.org/p/taskflow-mistral and see if we can align somehow Sure. (I hope it's not to late to do this, Never late IMO. seeing that there appears to be a lot of resistance from the mistral community to change. Could you please let us know how you made this conclusion? I’m frankly surprised.. It’s not just my curiosity or something, I want to keep improving in what I’m doing. So maybe its just my misinterpretation or miscommunication, but certain discussions like @ http://tinyurl.com/m395o2r<http://lists.openstack.org/pipermail/openstack-dev/2014-March/029983.html> (mention of 'inalienable parts of Mistral', 'Joshua propose detailed Mistral design based on TaskFlow') seem to be what causes me to think that what mistral has been making is more of a POC and is actually an implementation. To me that means the mistral project is already way past the POC mode (imho POC's are meant to explore concepts, then be thrown away and reimplemented as a real project, is mistral planning doing this, throwing way the POC and rewriting it as a non-POC using the ideas learned from the POC?)[http://tinyurl.com/lbz293s]. Generally, just to clarify the situation let me provide our vision of what we’re doing at the very high level. As mentioned many times, we’re still building a PoC. Yes, it turned out to take longer which is totally fine since we’ve done a lot of research, lots of coding exercises, talks, discussions with our customers. We’ve involved several new contributors from two different companies, they have their requirements and use cases too. We’ve gathered a lot of specific requirements to what should be a workflow engine. This all was the exact intention of that phase of the project: understand better what we should build. If you look at Mistral list of blueprints you’ll see around 40 of them where 80-90% of them come from real needs of real projects. And not everything is still captured in BPs because something is still not shaped well enough in our minds. Thought #1: In POC we’ve been concentrating on use cases and requirements. Implementation has been secondary. Sure that’s fine that’s what a POC is for. See above. TaskFlow or anything else just hasn’t mattered a lot so far. But, at the same time, I want to remind that in December we tried to use TaskFlow to implement the very basic functionality in Mistral (only dependency based model). Honestly, we failed to produce a result that we would be satisfied with since TaskFlow lacked, for example, the ability to run tasks in an asynchronous manner. This was not a problem at all, this is the real world. So I created a BP to address that problem in TaskFlow ([0]). So we decided to proceed with it with an intent to rejoin later. And may be even the most important reason not to use TaskFlow was that we did want to have a clear research. We found that less productive to try to build a project around an existing library than concentrating on use cases and high-level requirements. From my experience, it never works well since in your thinking you always stick to limitations of that lib and assumptions made in it. For the 'asynchronous manner' discussion see http://tinyurl.com/n3v9lt8; I'm still not sure why u would want to make is_sync/is_async a primitive concept in a workflow system, shouldn't this be only up to the entity running the workflow to decide? Why is a task allowed to be sync/async, that has major side-effects for state-persistence, resumption (and to me is a incorrect abstraction to provide) and general workflow execution control, I'd be very careful with this (which is why I am hesitant to add it without much much more discussion). So we actually talked to people a lot (including Josh) and provided this reasoning when this question raised again. Reaction was nearly always positive and made a lot of sense to customers and developers. Thought #2: A library shouldn't drive a project where it’s used. To me this assumes said library is fixed in stone, can't be changed, and can't be evolved. If a library is 'dead/vaporware' then sure, I would 100% agree with this, but all libraries in openstack do not fit into the later category; and those libraries can be evolved/developed/improved. As a community I think it is our goal to grow the libraries in the communi
Re: [openstack-dev] [Mistral] Actions design BP
On 18 Mar 2014, at 01:32, Joshua Harlow wrote: > To further this lets continue working on > https://etherpad.openstack.org/p/taskflow-mistral and see if we can align > somehow Sure. > (I hope it's not to late to do this, Never late IMO. > seeing that there appears to be a lot of resistance from the mistral > community to change. Could you please let us know how you made this conclusion? I’m frankly surprised.. It’s not just my curiosity or something, I want to keep improving in what I’m doing. Generally, just to clarify the situation let me provide our vision of what we’re doing at the very high level. As mentioned many times, we’re still building a PoC. Yes, it turned out to take longer which is totally fine since we’ve done a lot of research, lots of coding exercises, talks, discussions with our customers. We’ve involved several new contributors from two different companies, they have their requirements and use cases too. We’ve gathered a lot of specific requirements to what should be a workflow engine. This all was the exact intention of that phase of the project: understand better what we should build. If you look at Mistral list of blueprints you’ll see around 40 of them where 80-90% of them come from real needs of real projects. And not everything is still captured in BPs because something is still not shaped well enough in our minds. Thought #1: In POC we’ve been concentrating on use cases and requirements. Implementation has been secondary. TaskFlow or anything else just hasn’t mattered a lot so far. But, at the same time, I want to remind that in December we tried to use TaskFlow to implement the very basic functionality in Mistral (only dependency based model). Honestly, we failed to produce a result that we would be satisfied with since TaskFlow lacked, for example, the ability to run tasks in an asynchronous manner. This was not a problem at all, this is the real world. So I created a BP to address that problem in TaskFlow ([0]). So we decided to proceed with it with an intent to rejoin later. And may be even the most important reason not to use TaskFlow was that we did want to have a clear research. We found that less productive to try to build a project around an existing library than concentrating on use cases and high-level requirements. From my experience, it never works well since in your thinking you always stick to limitations of that lib and assumptions made in it. So we actually talked to people a lot (including Josh) and provided this reasoning when this question raised again. Reaction was nearly always positive and made a lot of sense to customers and developers. Thought #2: A library shouldn't drive a project where it’s used. Project needs should define the requirements to a library. Even being adopted in lots of places (this is apparently not true for TaskFlow right now which is fully understandable, a pretty young project as well) a library is just one of many tools used within a project. But not vice versa. Thought #3. We 100% admit we’re not ready for incubation right now. We prepared an incubation request already with most of the formal requirements met by the project. And even thought the interest to Mistral is serious (at least 5-6 projects intent to use it, one already started playing and prototyping with Mistral) we want to be honest with the community in that we’re not really ready for incubation since even our own solid understanding of core requirements and API/DSL is only on its way. This is just one more argument that spending significant time on thinking how to fit TaskFlow into our unstable Mistral vision is not affordable for us at the moment. Despite of that, last week we started spending that time after Josh started bombing us with his concerns :). If that’s all harsh, sorry. We’re interested in keeping in touch with Josh and others. [0] https://blueprints.launchpad.net/taskflow/+spec/async-tasks Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Actions design BP
Since the uploading of JS is an interesting concept, let me just share a little about another service at Y! that has done this. http://developer.yahoo.com/yql/guide/yql-execute-chapter.html YQL (although closed-source) has exposed this ability, so its entirely possible to do (although it likely won't be running in python). YQL itself is slightly similar to a DSL also (its DSL is similar to SQL). The nice thing about using something like JS (to continue along this idea, even if nobody wants to actually implement this) is that things like rhino<http://en.wikipedia.org/wiki/Rhino_%28JavaScript_engine%29> do provide execution limits (at the instruction level). Of course this would potentially bring in java (I am guessing node.js can do something similar as rhino). Anyways… To further this lets continue working on https://etherpad.openstack.org/p/taskflow-mistral and see if we can align somehow (I hope it's not to late to do this, seeing that there appears to be a lot of resistance from the mistral community to change). But I agree with clint, and hope that we can have a healthy collaboration as a community instead of being in competing silos (which is not healthy). -Josh From: Clint Byrum mailto:cl...@fewbar.com>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Date: Monday, March 17, 2014 at 9:41 AM To: openstack-dev mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack-dev] [Mistral] Actions design BP Excerpts from Renat Akhmerov's message of 2014-03-17 07:35:02 -0700: On 16 Mar 2014, at 19:05, Clint Byrum mailto:cl...@fewbar.com>> wrote: > From my perspective, as somebody who is considering Mistral for some > important things, the fact that the taskflow people are not aligned with > the Mistral people tells me that Mistral is not what I thought it was: > taskflow for direct user consumption. Yes, it was just an initial idea we were trying to pursue. As we moved forward we understood it was too limited and had a lot of pitfalls. The reason is the key difference between library and service. Making something a service flips everything upside down in terms of requirements to this new system. The logical questions/implications are: If that’s a service why should its REST API be Python oriented? Bindings - yes (it gives a certain convenience) but that’s a different story.. A whole set of questions related to how we distribute Python-written tasks (especially in multi tenant environment, e.g. as a cloud hosted service): Dependencies Sandboxing Result serialisation etc. It becomes logical to be able to use it with non-python clients and external systems. Orientation to long-lived processes(workflows). Synchronous execution model no longer works well, instead de facto asynchronous event-based architecture fits better. I know it may sound too generic, it’s a whole different topic.. It becomes logical to provide more high-level capabilities rather than a library does (e.g. Horizon Dashboard. It would not be much of use if the service was based on Python). I assume Mistral is written in Python though, and so it should be using Taskflow for its own workflow. I understand though, that to run _user_ workflows you can't just expect them to upload python or always run python as Mistral wouldn't do much for them at that point. However, for the User<->Taskflow integration, I think Josh Harlow offered a really reasonable suggestion to that, which is instead of inventing a DSL (frankly, I'm pretty frustrated with the Heat DSL's we have, so I may be biased here), just embrace javascript or lua, which are designed to be embedded and executed in a less-than-trusted context. I think it would be a great story for users if Mistral worked like this: - Upload javascript expression of workflow, with external callouts for complicated things. - Run code that uses Mistral API to poll-for or subscribe-to notifications waiting for instructions from Mistral when it is supposed to run said external callouts, and feeds back data. I believe it’s not the full list. Despite of all this I’m not against of using TaskFlow in implementation, at least some pieces of it. My only call is: it should be really beneficial rather than bringing pain. So I’m ok if we keep aligning our capabilities, roadmaps, terminology and whatever else is needed. Btw, it would be really helpful if you could clarify what you meant “some important things”. Asking because I still feel like we could have much more input from potential users. Any information is appreciated. - Heat needs to move from being a single state machine (heat-engine owns all of the running tasks for a stack in one engine at a time) to a distributed state machine. Before we do that, we need to consider how Heat expresses workflow. If Mistral were a distributed workflow e
Re: [openstack-dev] [Mistral] Actions design BP
Excerpts from Renat Akhmerov's message of 2014-03-17 07:35:02 -0700: > > On 16 Mar 2014, at 19:05, Clint Byrum wrote: > > > From my perspective, as somebody who is considering Mistral for some > > important things, the fact that the taskflow people are not aligned with > > the Mistral people tells me that Mistral is not what I thought it was: > > taskflow for direct user consumption. > > Yes, it was just an initial idea we were trying to pursue. As we moved > forward we understood it was too limited and had a lot of pitfalls. The > reason is the key difference between library and service. Making something a > service flips everything upside down in terms of requirements to this new > system. The logical questions/implications are: > If that’s a service why should its REST API be Python oriented? Bindings - > yes (it gives a certain convenience) but that’s a different story.. > A whole set of questions related to how we distribute Python-written tasks > (especially in multi tenant environment, e.g. as a cloud hosted service): > Dependencies > Sandboxing > Result serialisation > etc. > It becomes logical to be able to use it with non-python clients and external > systems. > Orientation to long-lived processes(workflows). Synchronous execution model > no longer works well, instead de facto asynchronous event-based architecture > fits better. I know it may sound too generic, it’s a whole different topic.. > It becomes logical to provide more high-level capabilities rather than a > library does (e.g. Horizon Dashboard. It would not be much of use if the > service was based on Python). > I assume Mistral is written in Python though, and so it should be using Taskflow for its own workflow. I understand though, that to run _user_ workflows you can't just expect them to upload python or always run python as Mistral wouldn't do much for them at that point. However, for the User<->Taskflow integration, I think Josh Harlow offered a really reasonable suggestion to that, which is instead of inventing a DSL (frankly, I'm pretty frustrated with the Heat DSL's we have, so I may be biased here), just embrace javascript or lua, which are designed to be embedded and executed in a less-than-trusted context. I think it would be a great story for users if Mistral worked like this: - Upload javascript expression of workflow, with external callouts for complicated things. - Run code that uses Mistral API to poll-for or subscribe-to notifications waiting for instructions from Mistral when it is supposed to run said external callouts, and feeds back data. > I believe it’s not the full list. > > Despite of all this I’m not against of using TaskFlow in implementation, at > least some pieces of it. My only call is: it should be really beneficial > rather than bringing pain. So I’m ok if we keep aligning our capabilities, > roadmaps, terminology and whatever else is needed. > > Btw, it would be really helpful if you could clarify what you meant “some > important things”. Asking because I still feel like we could have much more > input from potential users. Any information is appreciated. > - Heat needs to move from being a single state machine (heat-engine owns all of the running tasks for a stack in one engine at a time) to a distributed state machine. Before we do that, we need to consider how Heat expresses workflow. If Mistral were a distributed workflow engine, it would make a lot of sense for Heat to make use of it for this purpose. - TripleO deploys a number of things that need distributed workflow. I think at this point the people involved with that are looking more at lower level tools like RAFT and Concoord. But once the distributed state machine is settled, there will be a need to express the distributed workflow. I'm disinclined to diverge from Taskflow, even though I am quite inclined to embrace API's. > > So, while it may be annoying to have your "day to day project > > activities" questioned, it is pretty core to the discussion considering > > that this suggests several things that diverge from taskflow's core > > model. > > Np. I still keep finding things/rules in OpenStack that I’m not really aware > of or didn’t get used to yet. If that’s not how it should be done in OS then > it’s fine. I'm sorry if the message was harsh, but I see this happening a lot. I don't think it is a rule. I think the principle here is to collaborate on things that should be aligned. If Taskflow doesn't do what you need it to do, then I suggest _fixing that_ rather than writing a private implementation. It will make life better for all users of taskflow and it will keep Mistral extremely simple, which will help with adoption by operators _and_ users. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Actions design BP
On 16 Mar 2014, at 19:05, Clint Byrum wrote: > From my perspective, as somebody who is considering Mistral for some > important things, the fact that the taskflow people are not aligned with > the Mistral people tells me that Mistral is not what I thought it was: > taskflow for direct user consumption. Yes, it was just an initial idea we were trying to pursue. As we moved forward we understood it was too limited and had a lot of pitfalls. The reason is the key difference between library and service. Making something a service flips everything upside down in terms of requirements to this new system. The logical questions/implications are: If that’s a service why should its REST API be Python oriented? Bindings - yes (it gives a certain convenience) but that’s a different story.. A whole set of questions related to how we distribute Python-written tasks (especially in multi tenant environment, e.g. as a cloud hosted service): Dependencies Sandboxing Result serialisation etc. It becomes logical to be able to use it with non-python clients and external systems. Orientation to long-lived processes(workflows). Synchronous execution model no longer works well, instead de facto asynchronous event-based architecture fits better. I know it may sound too generic, it’s a whole different topic.. It becomes logical to provide more high-level capabilities rather than a library does (e.g. Horizon Dashboard. It would not be much of use if the service was based on Python). I believe it’s not the full list. Despite of all this I’m not against of using TaskFlow in implementation, at least some pieces of it. My only call is: it should be really beneficial rather than bringing pain. So I’m ok if we keep aligning our capabilities, roadmaps, terminology and whatever else is needed. Btw, it would be really helpful if you could clarify what you meant “some important things”. Asking because I still feel like we could have much more input from potential users. Any information is appreciated. > So, while it may be annoying to have your "day to day project > activities" questioned, it is pretty core to the discussion considering > that this suggests several things that diverge from taskflow's core > model. Np. I still keep finding things/rules in OpenStack that I’m not really aware of or didn’t get used to yet. If that’s not how it should be done in OS then it’s fine.___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Actions design BP
Excerpts from Renat Akhmerov's message of 2014-03-13 01:55:46 -0700: > Joshua, > > Thanks for your interest and feedback. > > I believe you were able to deliver your message already, we definitely hear > you. So no need to the same stuff again and again ;) As I promised before, we > are now evaluating what’s been going on with TaskFlow for the last couple of > months and preparing our questions/concerns/suggestions on using it within > Mistral. But that’s not very easy and quick thing to do since there’s a bunch > of details to take into account, especially given the fact that Mistral > codebase has become much more solid and Mistral itself now has lots of > requirements dictated by its use cases and roadmap vision. So patience would > be really appreciated here. > > If you don’t mind I would prefer to discuss things like that in separate > threads, not in threads devoted to daily project activities. So that we can > split our conceptual discussions and current work that’s going on according > to our plans. Otherwise we have a risk to make spaghetti out of our ML > threads. > From my perspective, as somebody who is considering Mistral for some important things, the fact that the taskflow people are not aligned with the Mistral people tells me that Mistral is not what I thought it was: taskflow for direct user consumption. So, while it may be annoying to have your "day to day project activities" questioned, it is pretty core to the discussion considering that this suggests several things that diverge from taskflow's core model. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Actions design BP
So this is an interesting question. Let me explain why I think exposing is_sync is actually relatively dangerous and breaks the task oriented model (at least taskflows view of it). First let me explain a little bit of what happens in taskflow to create and execute things. 1. User X creates objects derived from task<https://github.com/openstack/taskflow/blob/master/taskflow/task.py#L31> base class that act as the primitives that execute and revert (nothing here specifies that they are synchronous or not, they just do some piece of work and return some results and have the ability to undo those pieces of work). 2. User X combines links tasks (and also patterns themselves) together using taskflows patterns<https://github.com/openstack/taskflow/tree/master/taskflow/patterns> (currently a small set, but could get bigger as needed) to form a larger set of combined work (A -> B -> C for example), these patterns support nesting so [A] in the example can itself expand into a something like (E,F,G tasks) and so on. Lets call the result of this linking the Y object. NOTE: At this point there is still not is_sync or is_async, since here we are just defining ordering constraints that should be enforced at runtime. 3. User gives Y object to engine<https://github.com/openstack/taskflow/tree/master/taskflow/engines> in taskflow, providing the engine persistence<https://github.com/openstack/taskflow/tree/master/taskflow/persistence> model/backend it wants to use (for storing intermediate results, for saving state changes) and tells the engine to execute(). At this point the engine will locate all tasks that have no dependencies on other tasks and start running them asynchronously (how this works depends on the engine type selected), at this point the tasks that can execute begin executing (and they have the potential to signal to others there current progress via the update_progress<https://github.com/openstack/taskflow/blob/master/taskflow/task.py#L78> method (engines support a concept of listeners<https://github.com/openstack/taskflow/tree/master/taskflow/listeners> that can be attached to engines to allow external entities to be informed of progress updates, state changes…). This process repeats until the workflow has completed or it fails (in which case revert() methods start to be called and each task is given a chance to undo whatever it has created, in the near future there will be aways to alter how this reversion happens with a concept of retry_controllers<https://review.openstack.org/#/c/71621/>). TLDR: So in general u could say that all tasks are unaware that they are running async/sync and in the above model it is up to the engine type to determine how things are ran (since the engine is the controller that actually runs all tasks, making sure the the ordering constraints established in step #2 are retained). To me this kind of disconnection (not allowing a task to specify it's async/sync) is useful and helps retain sanity. In a way it seems against the task model to have tasks provide this kind of information (leaky abstraction…) and at least in the taskflow view makes tasks have to much control over how they are executed (why should a task care?). It also completely alters the state diagram<https://wiki.openstack.org/wiki/File:Tf_task_state_diagram.png> that is persisted and used for resumption when this information is allowed to be specified by a task. What does it mean for a task to be RUNNING but have the task continues to run asynchronously? In a way isn't this already what the engine (at least in taskflow) is doing internally anyway (running everything asynchronously)? At least in taskflow all task are already running with executors<https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/executor.py> that return future/s that are waited upon (dependent tasks can not run until the future of a predecessor task has completed). -Josh From: Renat Akhmerov mailto:rakhme...@mirantis.com>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Date: Friday, March 14, 2014 at 5:09 AM To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack-dev] [Mistral] Actions design BP - is_sync() - consider using an attribute instead - @mistral.async Well, I had an idea that it may depend on how a particular action instance is parameterized (in other words, a dynamic thing rather than static property). It would just give more flexibility - can we think of a way to unify sync and async actions from engine's standpoint? So that we don't special-case it in the engine? To be precise, engine has no knowledge about it. Only executor does and it has to but the difference is pretty small. In
Re: [openstack-dev] [Mistral] Actions design BP
On 14 Mar 2014, at 05:00, Dmitri Zimine wrote: > - Async actions: how do results of async action communicates back? > My understanding it it assumes that the remote system will call back to > mistral with action execution id, it's on the engine to call-back, and action > needs to let the engine know to expect call-back. Let's put the explanation > here. Yes. When I was writing this I thought it was a little bit out of the scope. Basically, in API there’s a method which can be used to deliver a result once it’s been calculated. And then engine does what’s needed to update task state and advance the workflow. Currently, it’s only possible to do it via REST API and we also have a requirement to do it via MQ (it can be configurable). > - is_sync() - consider using an attribute instead - @mistral.async Well, I had an idea that it may depend on how a particular action instance is parameterized (in other words, a dynamic thing rather than static property). It would just give more flexibility > - can we think of a way to unify sync and async actions from engine's > standpoint? So that we don't special-case it in the engine? To be precise, engine has no knowledge about it. Only executor does and it has to but the difference is pretty small. In case if action is sync it should just call the API method I mentioned above to deliver action result. When we finish our BPs related to oslo.messaging it’ll be working over it. > @ Joshua - does something similar exists in TaskFlow already? As far as I know, it doesn’t exist in TaskFlow but I may be wrong.. > - def dry_run() - maybe name "test", let's stress that this method should > return a representative sample output. Ok, I like that. Will change this part. > - Input - need a facility to declare, validate and list input parameters. > Like VALID_KEYS=['url', 'parameters''] , def validate(): > > - class HTTPAction(object): > def __init__(self, url, params, method, headers, body): > Not happy about declaring parameters explicitly. How about using * args > **kvargs, or 'parameters' dictionary? Yes, these two items are related. I think it’s a matter of responsibility distribution. Whether we want validation to happen in action factory or in action itself. I tend to agree with you on this one that action class should be responsible itself for validation. Let’s think more through that. > - DSL In-Place Declaration - I did minor edits in the section, please check. Ok. Thanks. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Actions design BP
Thanks Renat for a clear design summary, thanks Joshua for the questions, +1 to "let's move TaskFlow vs Mistral" discussion to separate thread, and my questions/comments on https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign below: - Async actions: how do results of async action communicates back? My understanding it it assumes that the remote system will call back to mistral with action execution id, it's on the engine to call-back, and action needs to let the engine know to expect call-back. Let's put the explanation here. - is_sync() - consider using an attribute instead - @mistral.async - can we think of a way to unify sync and async actions from engine's standpoint? So that we don't special-case it in the engine? @ Joshua - does something similar exists in TaskFlow already? - def dry_run() - maybe name "test", let's stress that this method should return a representative sample output. - Input - need a facility to declare, validate and list input parameters. Like VALID_KEYS=['url', 'parameters''] , def validate(): - class HTTPAction(object): def __init__(self, url, params, method, headers, body): Not happy about declaring parameters explicitly. How about using * args **kvargs, or 'parameters' dictionary? - DSL In-Place Declaration - I did minor edits in the section, please check. DZ. - On Mar 12, 2014, at 6:54 PM, Joshua Harlow wrote: > So taskflow has tasks, which seems comparable to actions? > > I guess I should get tired of asking but why recreate the same stuff ;) > > The questions listed: > > - Does action need to have revert() method along with run() method? > - How does action expose errors occurring during it's work? > > - In what form does action return a result? > > > And more @ https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign > > And quite a few others that haven't been mentioned (how does a action > retry? How does a action report partial progress? What's the > intertask/state persistence mechanism?) have been worked on by the > taskflow team for a while now... > > https://github.com/openstack/taskflow/blob/master/taskflow/task.py#L31 > (and others...) > > Anyways, I know mistral is still POC/pilot/prototype... but seems like > more duplicate worked that could just be avoided ;) > > -Josh > > -Original Message- > From: Renat Akhmerov > Reply-To: "OpenStack Development Mailing List (not for usage questions)" > > Date: Tuesday, March 11, 2014 at 11:32 PM > To: "OpenStack Development Mailing List (not for usage questions)" > > Subject: [openstack-dev] [Mistral] Actions design BP > >> Team, >> >> I started summarizing all the thoughts and ideas that we¹ve been >> discussing for a while regarding actions. The main driver for this work >> is that the system keeps evolving and we still don¹t have a comprehensive >> understanding of that part. Additionally, we keep getting a lot of >> requests and questions from our potential users which are related to >> actions (Œwill they be extensible?¹, Œwill they have dry-run feature?¹, >> Œwhat are the ways to configure and group them?¹ and so on and so forth). >> So although we¹re still in a Pilot phase we need to start this work in >> parallel. Even now lack of solid understanding of it creates a lot of >> problems in pilot development. >> >> I created a BP at launchpad [0] which has a reference to detailed >> specification [1]. It¹s still in progress but you could already leave >> your early feedback so that I don¹t go in a wrong direction too far. >> >> The highest priority now is still finishing the pilot so we shouldn¹t >> start implementing everything described in BP right now. However, some of >> the things have to be adjusted asap (like Action interface and the main >> implementation principles). >> >> [0]: >> https://blueprints.launchpad.net/mistral/+spec/mistral-actions-design >> [1]: https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign >> >> Renat Akhmerov >> @ Mirantis Inc. >> >> >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Actions design BP
Joshua, Thanks for your interest and feedback. I believe you were able to deliver your message already, we definitely hear you. So no need to the same stuff again and again ;) As I promised before, we are now evaluating what’s been going on with TaskFlow for the last couple of months and preparing our questions/concerns/suggestions on using it within Mistral. But that’s not very easy and quick thing to do since there’s a bunch of details to take into account, especially given the fact that Mistral codebase has become much more solid and Mistral itself now has lots of requirements dictated by its use cases and roadmap vision. So patience would be really appreciated here. If you don’t mind I would prefer to discuss things like that in separate threads, not in threads devoted to daily project activities. So that we can split our conceptual discussions and current work that’s going on according to our plans. Otherwise we have a risk to make spaghetti out of our ML threads. Thanks Renat Akhmerov @ Mirantis Inc. On 13 Mar 2014, at 08:54, Joshua Harlow wrote: > So taskflow has tasks, which seems comparable to actions? > > I guess I should get tired of asking but why recreate the same stuff ;) > > The questions listed: > > - Does action need to have revert() method along with run() method? > - How does action expose errors occurring during it's work? > > - In what form does action return a result? > > > And more @ https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign > > And quite a few others that haven't been mentioned (how does a action > retry? How does a action report partial progress? What's the > intertask/state persistence mechanism?) have been worked on by the > taskflow team for a while now... > > https://github.com/openstack/taskflow/blob/master/taskflow/task.py#L31 > (and others...) > > Anyways, I know mistral is still POC/pilot/prototype... but seems like > more duplicate worked that could just be avoided ;) > > -Josh > > -Original Message- > From: Renat Akhmerov > Reply-To: "OpenStack Development Mailing List (not for usage questions)" > > Date: Tuesday, March 11, 2014 at 11:32 PM > To: "OpenStack Development Mailing List (not for usage questions)" > > Subject: [openstack-dev] [Mistral] Actions design BP > >> Team, >> >> I started summarizing all the thoughts and ideas that we¹ve been >> discussing for a while regarding actions. The main driver for this work >> is that the system keeps evolving and we still don¹t have a comprehensive >> understanding of that part. Additionally, we keep getting a lot of >> requests and questions from our potential users which are related to >> actions (Œwill they be extensible?¹, Œwill they have dry-run feature?¹, >> Œwhat are the ways to configure and group them?¹ and so on and so forth). >> So although we¹re still in a Pilot phase we need to start this work in >> parallel. Even now lack of solid understanding of it creates a lot of >> problems in pilot development. >> >> I created a BP at launchpad [0] which has a reference to detailed >> specification [1]. It¹s still in progress but you could already leave >> your early feedback so that I don¹t go in a wrong direction too far. >> >> The highest priority now is still finishing the pilot so we shouldn¹t >> start implementing everything described in BP right now. However, some of >> the things have to be adjusted asap (like Action interface and the main >> implementation principles). >> >> [0]: >> https://blueprints.launchpad.net/mistral/+spec/mistral-actions-design >> [1]: https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign >> >> Renat Akhmerov >> @ Mirantis Inc. >> >> >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Actions design BP
"So no need to ask the same…” Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Actions design BP
So taskflow has tasks, which seems comparable to actions? I guess I should get tired of asking but why recreate the same stuff ;) The questions listed: - Does action need to have revert() method along with run() method? - How does action expose errors occurring during it's work? - In what form does action return a result? And more @ https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign And quite a few others that haven't been mentioned (how does a action retry? How does a action report partial progress? What's the intertask/state persistence mechanism?) have been worked on by the taskflow team for a while now... https://github.com/openstack/taskflow/blob/master/taskflow/task.py#L31 (and others...) Anyways, I know mistral is still POC/pilot/prototype... but seems like more duplicate worked that could just be avoided ;) -Josh -Original Message- From: Renat Akhmerov Reply-To: "OpenStack Development Mailing List (not for usage questions)" Date: Tuesday, March 11, 2014 at 11:32 PM To: "OpenStack Development Mailing List (not for usage questions)" Subject: [openstack-dev] [Mistral] Actions design BP >Team, > >I started summarizing all the thoughts and ideas that we¹ve been >discussing for a while regarding actions. The main driver for this work >is that the system keeps evolving and we still don¹t have a comprehensive >understanding of that part. Additionally, we keep getting a lot of >requests and questions from our potential users which are related to >actions (Œwill they be extensible?¹, Œwill they have dry-run feature?¹, >Œwhat are the ways to configure and group them?¹ and so on and so forth). >So although we¹re still in a Pilot phase we need to start this work in >parallel. Even now lack of solid understanding of it creates a lot of >problems in pilot development. > >I created a BP at launchpad [0] which has a reference to detailed >specification [1]. It¹s still in progress but you could already leave >your early feedback so that I don¹t go in a wrong direction too far. > >The highest priority now is still finishing the pilot so we shouldn¹t >start implementing everything described in BP right now. However, some of >the things have to be adjusted asap (like Action interface and the main >implementation principles). > >[0]: >https://blueprints.launchpad.net/mistral/+spec/mistral-actions-design >[1]: https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign > >Renat Akhmerov >@ Mirantis Inc. > > > > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Actions design BP
Team, I started summarizing all the thoughts and ideas that we’ve been discussing for a while regarding actions. The main driver for this work is that the system keeps evolving and we still don’t have a comprehensive understanding of that part. Additionally, we keep getting a lot of requests and questions from our potential users which are related to actions (‘will they be extensible?’, ‘will they have dry-run feature?’, ‘what are the ways to configure and group them?’ and so on and so forth). So although we’re still in a Pilot phase we need to start this work in parallel. Even now lack of solid understanding of it creates a lot of problems in pilot development. I created a BP at launchpad [0] which has a reference to detailed specification [1]. It’s still in progress but you could already leave your early feedback so that I don’t go in a wrong direction too far. The highest priority now is still finishing the pilot so we shouldn’t start implementing everything described in BP right now. However, some of the things have to be adjusted asap (like Action interface and the main implementation principles). [0]: https://blueprints.launchpad.net/mistral/+spec/mistral-actions-design [1]: https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
