While I appreciate the many ideas being discussed here (some of which we've explored previously and agreed to continue exploring), there is a fundamental difference vs. what I propose in that spec. I believe that what I'm proposing will be achievable without any significant visible changes in the API -- no new API end points or resources, and the client interaction will be nearly the same. A few status codes may be different in certain circumstances -- but it will not require a new major version of the REST API. And it solves a scalability and stability problem that folks are encountering today. (It seems my spec didn't describe those problems well enough -- I'm updating it now.)
Cheers, Devananda On Wed, May 28, 2014 at 10:14 AM, Maksym Lobur <mlo...@mirantis.com> wrote: > BTW a very similar discussion is going in Neutron community right now, > please find a thread under the *[openstack-dev] [Neutron] Introducing > task oriented workflows* label. > > Best regards, > Max Lobur, > Python Developer, Mirantis, Inc. > > Mobile: +38 (093) 665 14 28 > Skype: max_lobur > > 38, Lenina ave. Kharkov, Ukraine > www.mirantis.com > www.mirantis.ru > > > On Wed, May 28, 2014 at 6:56 PM, Maksym Lobur <mlo...@mirantis.com> wrote: > >> Hi All, >> >> You've raised a good discussion, something similar already was started >> back in february. Could someone please find the long etherpad with >> discussion between Deva and Lifeless, as I recall most of the points >> mentioned above have a good comments there. >> >> Up to this point I have the only idea how to elegantly address these >> problems. This is a tasks concept and probably a scheduler service, which >> not necessarily should be separate from the API at the moment (Finally we >> already have a hash ring on the api side which is a kind of scheduler >> right?) It was already proposed earlier, but I would like to try to fit all >> these issues into this concept. >> >> >>> 1. "Executability" >>> We need to make sure that request can be theoretically executed, >>> which includes: >>> a) Validating request body >>> >> >> We cannot validate everything on the API side, relying on the fact that >> DB state is actual is not a good idea, especially under heavy load. >> >> In tasks concept we could assume that all the requests are executable, >> and do not perform any validation in the API thread at all. Instead of this >> the API will just create a task and return it's ID to the user. Task >> scheduler may perform some minor validations before the task is queued or >> started for convenience, but they should be duplicated inside task body >> because there is an arbitrary time between queuing up and start >> ((c)lifeless). I assume the scheduler will have it's own thread or even >> process. The user will need to poke received ID to know the current state >> of his submission. >> >> >>> b) For each of entities (e.g. nodes) touched, check that they are >>> available >>> at the moment (at least exist). >>> This is arguable, as checking for entity existence requires going to >>> DB. >> >> >> Same here, DB round trip is a potential block, therefore this will be >> done inside task (after it's queued and started) and will not affect the >> API. The user will just observe the task state by poking the API (or using >> callback as an option). >> >> 2. Appropriate state >>> For each entity in question, ensure that it's either in a proper state >>> or >>> moving to a proper state. >>> It would help avoid users e.g. setting deploy twice on the same node >>> It will still require some kind of NodeInAWrongStateError, but we won't >>> necessary need a client retry on this one. >>> Allowing the entity to be _moving_ to appropriate state gives us a >>> problem: >>> Imagine OP1 was running and OP2 got scheduled, hoping that OP1 will come >>> to desired state. What if OP1 fails? What if conductor, doing OP1 >>> crashes? >> >> >> Let's say OP1 and OP2 are two separate tasks. Each one have the initial >> state validation inside it's body. Once OP2 gets its turn it will perform >> validation and fail, which looks reasonable to me. >> >> >>> Similar problem with checking node state. >>> Imagine we schedule OP2 while we had OP1 - regular checking node state. >>> OP1 discovers that node is actually absent and puts it to maintenance >>> state. >>> What to do with OP2? >> >> >> The task will fail once it get it's turn. >> >> >>> b) Can we make client wait for the results of periodic check? >>> That is, wait for OP1 _before scheduling_ OP2? >> >> >> We will just schedule the task and the user will observe its progress, >> once OP1 is finished and OP2 started - he will see a fail. >> >> >>> 3. Status feedback >>> People would like to know, how things are going with their task. >>> What they know is that their request was scheduled. Options: >>> a) Poll: return some REQUEST_ID and expect users to poll some endpoint. >>> Pros: >>> - Should be easy to implement >>> Cons: >>> - Requires persistent storage for tasks. Does AMQP allow to do this >>> kinds >>> of queries? If not, we'll need to duplicate tasks in DB. >>> - Increased load on API instances and DB >>> >> >> Exactly described the tasks concept :) >> >> >>> b) Callback: take endpoint, call it once task is done/fails. >>> Pros: >>> - Less load on both client and server >>> - Answer exactly when it's ready >>> Cons: >>> - Will not work for cli and similar >>> - If conductor crashes, there will be no callback. >> >> >> Add to Cons: >> - Callback is not reliable since it may get lost. >> We should have an ability to poke anyway, though I see a great benefit >> from implementing a callbacks - to decrease API load. >> >> >>> 4. Debugging consideration >>> a) This is an open question: how to debug, if we have a lot of requests >>> and something went wrong? >>> >> >> We will be able to see the queue state (btw what about security here, >> should the user be able to see all the tasks, or just his ones, or all but >> others with hidden details). >> >> >>> b) One more thing to consider: how to make command like `node-show` >>> aware of >>> scheduled transitioning, so that people don't try operations that are >>> doomed to failure. >> >> >> node-show will always show current state of the node, though we may check >> if there are any tasks queued or going, which will change the state. If any >> - add a notification to the response. >> >> >>> 5. Performance considerations >>> a) With async approach, users will be able to schedule nearly unlimited >>> number of tasks, thus essentially blocking work of Ironic, without >>> any >>> signs of the problem (at least for some time). >>> I think there are 2 common answers to this problem: >>> - Request throttling: disallow user to make too many requests in some >>> amount of time. Send them 503 with Retry-After header set. >> >> >> Can this be achieved by some web-server settings? Looks like a typical >> problem. >> >> >>> - Queue management: watch queue length, deny new requests if it's too >>> large. >>> >> >> Yes, I really like the limited queue size idea. Please see my comments in >> the spec. >> Also, if we have a tasks and the queue, we could merge similar tasks >> >> >>> b) State framework from (2), if invented, can become a bottleneck as >>> well. >>> Especially with polling approach. >> >> >> True. >> If we have tasks, all the node actions will be done through them. We can >> synchronise node state with DB only during the task, and remove periodic >> syncs. Off-course someone may go and turn off the node, in this case the >> Ironic will lie about the node state until some task is executed on this >> node, which may be suitable behaviour. Otherwise rare periodic syncs may >> work as well. >> >> >>> 6. Usability considerations >>> a) People will be unaware, when and whether their request is going to be >>> finished. As there will be tempted to retry, we may get flooded by >>> duplicates. I would suggest at least make it possible to request >>> canceling >>> any task (which will be possible only if it is not started yet, >>> obviously). >>> >> >> Since we will have a limited number of kinds of tasks, we could calculate >> some estimates basing on previous similar tasks. Looks like an improvement >> for a distant future. In the end I wouldn't want Ironic to perform >> estimates like windows's copy-paste dialog :) >> >> Tasks may be easily interrupted while they are in a queue. But if it's >> already started - there's a separate dicsussion >> https://blueprints.launchpad.net/ironic/+spec/make-tasks-interruptible(I'm >> going to port this bp to the specs repo in some time) >> >> >>> b) We should try to avoid scheduling contradictive requests. >>> >> >> A task scheduler responsibility: this is basically a state check before >> task is scheduled, and it should be done one more time once the task is >> started, as mentioned above. >> >> >>> c) Can we somehow detect duplicated requests and ignore them? >>> E.g. we won't want user to make 2-3-4 reboots in a row just because >>> the user >>> was not patient enough. >> >> >> Queue similar tasks. All the users will be pointed to the similar task >> resource, or maybe to a different resources which tied to the same >> conductor action. >> >> >>> Best regards, >>> Max Lobur, >>> Python Developer, Mirantis, Inc. >>> Mobile: +38 (093) 665 14 28 >>> Skype: max_lobur >>> 38, Lenina ave. Kharkov, Ukraine >>> www.mirantis.com >>> www.mirantis.ru >>> >>> >>> On Wed, May 28, 2014 at 5:10 PM, Lucas Alvares Gomes < >>> lucasago...@gmail.com> wrote: >>> On Wed, May 28, 2014 at 2:02 PM, Dmitry Tantsur <dtant...@redhat.com> >>> wrote: >>> > Hi Ironic folks, hi Devananda! >>> > >>> > I'd like to share with you my thoughts on asynchronous API, which is >>> > spec https://review.openstack.org/#/c/94923 >>> > First I was planned this as comments to the review, but it proved to be >>> > much larger, so I post it for discussion on ML. >>> > >>> > Here is list of different consideration, I'd like to take into account >>> > when prototyping async support, some are reflected in spec already, >>> some >>> > are from my and other's comments: >>> > >>> > 1. "Executability" >>> > We need to make sure that request can be theoretically executed, >>> > which includes: >>> > a) Validating request body >>> > b) For each of entities (e.g. nodes) touched, check that they are >>> > available >>> > at the moment (at least exist). >>> > This is arguable, as checking for entity existence requires going to >>> > DB. >>> >>> > >>> > 2. Appropriate state >>> > For each entity in question, ensure that it's either in a proper state >>> > or >>> > moving to a proper state. >>> > It would help avoid users e.g. setting deploy twice on the same node >>> > It will still require some kind of NodeInAWrongStateError, but we won't >>> > necessary need a client retry on this one. >>> > >>> > Allowing the entity to be _moving_ to appropriate state gives us a >>> > problem: >>> > Imagine OP1 was running and OP2 got scheduled, hoping that OP1 will >>> come >>> > to desired state. What if OP1 fails? What if conductor, doing OP1 >>> > crashes? >>> > That's why we may want to approve only operations on entities that do >>> > not >>> > undergo state changes. What do you think? >>> > >>> > Similar problem with checking node state. >>> > Imagine we schedule OP2 while we had OP1 - regular checking node state. >>> > OP1 discovers that node is actually absent and puts it to maintenance >>> > state. >>> > What to do with OP2? >>> > a) Obvious answer is to fail it >>> > b) Can we make client wait for the results of periodic check? >>> > That is, wait for OP1 _before scheduling_ OP2? >>> > >>> > Anyway, this point requires some state framework, that knows about >>> > states, >>> > transitions, actions and their compatibility with each other. >>> For {power, provision} state changes should we queue the requests? We >>> may want to only accept 1 request to change the state per time, if a >>> second request comes when there's another state change mid-operation >>> we may just return 409 (Conflict) to indicate that a state change is >>> already in progress. This is similar of what we have today but instead >>> of checking the node lock and states on the conductor side the API >>> service could do it, since it's on the DB. >>> > >>> > 3. Status feedback >>> > People would like to know, how things are going with their task. >>> > What they know is that their request was scheduled. Options: >>> > a) Poll: return some REQUEST_ID and expect users to poll some endpoint. >>> > Pros: >>> > - Should be easy to implement >>> > Cons: >>> > - Requires persistent storage for tasks. Does AMQP allow to do this >>> > kinds >>> > of queries? If not, we'll need to duplicate tasks in DB. >>> > - Increased load on API instances and DB >>> > b) Callback: take endpoint, call it once task is done/fails. >>> > Pros: >>> > - Less load on both client and server >>> > - Answer exactly when it's ready >>> > Cons: >>> > - Will not work for cli and similar >>> > - If conductor crashes, there will be no callback. >>> > >>> > Seems like we'd want both (a) and (b) to comply with current needs. >>> +1, we could allow pooling by default (like checking >>> nodes/<uuid>/states to know the current and target state of the node) >>> but we may also want to include a callback parameter that users could >>> use to input a URL that the conductor will call out as soon as the >>> operation is finished. So if the callback URl exists, the conductor >>> will submit a POST request to that URL with some data structure >>> identifying the operation and the current state. >>> > >>> > If we have a state framework from (2), we can also add notifications to >>> > it. >>> > >>> > 4. Debugging consideration >>> > a) This is an open question: how to debug, if we have a lot of requests >>> > and something went wrong? >>> > b) One more thing to consider: how to make command like `node-show` >>> > aware of >>> > scheduled transitioning, so that people don't try operations that >>> are >>> > doomed to failure. >>> > >>> > 5. Performance considerations >>> > a) With async approach, users will be able to schedule nearly unlimited >>> > number of tasks, thus essentially blocking work of Ironic, without >>> > any >>> > signs of the problem (at least for some time). >>> > I think there are 2 common answers to this problem: >>> > - Request throttling: disallow user to make too many requests in >>> some >>> > amount of time. Send them 503 with Retry-After header set. >>> > - Queue management: watch queue length, deny new requests if it's >>> too >>> > large. >>> > This means actually getting back error 503 and will require retrying >>> > again! >>> > At least it will be exceptional case, and won't affect Tempest >>> run... >>> > b) State framework from (2), if invented, can become a bottleneck as >>> > well. >>> > Especially with polling approach. >>> > >>> > 6. Usability considerations >>> > a) People will be unaware, when and whether their request is going to >>> be >>> > finished. As there will be tempted to retry, we may get flooded by >>> > duplicates. I would suggest at least make it possible to request >>> > canceling >>> > any task (which will be possible only if it is not started yet, >>> > obviously). >>> > b) We should try to avoid scheduling contradictive requests. >>> > c) Can we somehow detect duplicated requests and ignore them? >>> > E.g. we won't want user to make 2-3-4 reboots in a row just because >>> > the user >>> > was not patient enough. >>> > >>> > ------ >>> > >>> > Possible takeaways from this letter: >>> > - We'll need at least throttling to avoid DoS >>> > - We'll still need handling of 503 error, though it should not happen >>> > under >>> > normal conditions >>> > - Think about state framework that unifies all this complex logic with >>> > features: >>> > * Track entities, their states and actions on entities >>> > * Check whether new action is compatible with states of entities it >>> > touches >>> > and with other ongoing and scheduled actions on these entities. >>> > * Handle notifications for finished and failed actions by providing >>> > both >>> > pull and push approaches. >>> > * Track whether started action is still executed, perform error >>> > notification, >>> > if not. >>> > * HA and high performance >>> > - Think about policies for corner cases >>> > - Think, how we can make a user aware of what is going on with both >>> > request >>> > and entity that some requests may touch. Also consider canceling >>> > requests. >>> > >>> > Please let me know, what you think. >>> > >>> > Dmitry. >>> > >>> > >>> > _______________________________________________ >>> > OpenStack-dev mailing list >>> > OpenStack-dev@lists.openstack.org >>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> _______________________________________________ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> + 1 >>> >> >> > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev