Cool, feel free to jump on IRC[1] if u need in-person details/questions or other (consultation is free, haha).
[1] irc://chat.freenode.net/openstack-state-management -Josh -----Original Message----- From: Eoghan Glynn <[email protected]> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <[email protected]> Date: Tuesday, June 17, 2014 at 3:11 PM To: "OpenStack Development Mailing List (not for usage questions)" <[email protected]> Subject: Re: [openstack-dev] [taskflow] Recommendations for the granularity of tasks and their "stickiness" to workers > >Thanks to Joshua and Sandy for the clarifications on the finer >points of taskflow features and usage. > >I'll consume the docco linked, and will be back to annoy ye with >further questions :) > >Cheers, >Eoghan > > >> Back on the distributed subject (since this deserves a different email), >> >> In the newest taskflow release (0.3.x) we have 2 mechanisms for >> distribution outside of a process. >> >> One is the job/jobboard[1] & conductor[2] concepts, >> >> These concepts allow for atomic ownership of a 'job' and conductors act >>as >> one way (not the only way) to perform the work in a distributed manner >> (conductors consume and perform work). Conductors are pretty new so if >> bugs are found please let us know. Jobs have some unique properties >>that I >> think make them attractive[3]. >> >> The second method here is what is called the W.B.E. (the worker based >> engine), >> >> This is different in that it is a distribution at the engine[4] layer >>(not >> at the job layer), this engine allows for running tasks on remote >>workers >> (and it can be used in combination with the job concept/layer). >> Documentation for this can be found @ >> http://docs.openstack.org/developer/taskflow/workers.html; this WBE is >> under development, it does work, but it does have a few limitations that >> still need addressing (see docs link). >> >> So that¹s what exists currently (not just in-process things), >> >> [1] http://docs.openstack.org/developer/taskflow/jobs.html >> [2] http://docs.openstack.org/developer/taskflow/conductors.html >> [3] http://docs.openstack.org/developer/taskflow/jobs.html#features >> [4] http://docs.openstack.org/developer/taskflow/engines.html >> >> -----Original Message----- >> From: Sandy Walsh <[email protected]> >> Reply-To: "OpenStack Development Mailing List (not for usage questions)" >> <[email protected]> >> Date: Tuesday, June 17, 2014 at 5:33 AM >> To: "OpenStack Development Mailing List (not for usage questions)" >> <[email protected]> >> Subject: Re: [openstack-dev] [taskflow] Recommendations for the >> granularity of tasks and their "stickiness" to workers >> >> >On 6/17/2014 7:04 AM, Eoghan Glynn wrote: >> >> Folks, >> >> >> >> A question for the taskflow ninjas. >> >> >> >> Any thoughts on best practice WRT $subject? >> >> >> >> Specifically I have in mind this ceilometer review[1] which adopts >> >> the approach of using very fine-grained tasks (at the level of an >> >> individual alarm evaluation) combined with short-term assignments >> >> to individual workers. >> >> >> >> But I'm also thinking of future potential usage of taskflow within >> >> ceilometer, to support partitioning of work over a scaled-out array >> >> of central agents. >> >> >> >> Does taskflow also naturally support a model whereby more chunky >> >> tasks (possibly including ongoing periodic work) are assigned to >> >> workers in a stickier fashion, such that re-balancing of workload >> >> can easily be triggered when a change is detected in the pool of >> >> available workers? >> > >> >I don't think taskflow today is really focused on load balancing of >> >tasks. Something like gearman [1] might be better suited in the near >>term? >> > >> >My understanding is that taskflow is really focused on in-process tasks >> >(with retry, restart, etc) and later will support distributed tasks. >>But >> >my data could be stale too. (jharlow?) >> > >> >Even still, the decision of smaller tasks vs. chunky ones really comes >> >down to how much work you want to re-do if there is a failure. I've >>seen >> >some uses of taskflow where the breakdown of tasks seemed artificially >> >small. Meaning, the overhead of going back to the library on an >> >undo/rewind is greater than the undo itself. >> > >> >-S >> > >> >[1] http://gearman.org/ >> > >> > >> >_______________________________________________ >> >OpenStack-dev mailing list >> >[email protected] >> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> _______________________________________________ >> OpenStack-dev mailing list >> [email protected] >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >_______________________________________________ >OpenStack-dev mailing list >[email protected] >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
