Back on the distributed subject (since this deserves a different email), In the newest taskflow release (0.3.x) we have 2 mechanisms for distribution outside of a process.
One is the job/jobboard[1] & conductor[2] concepts, These concepts allow for atomic ownership of a 'job' and conductors act as one way (not the only way) to perform the work in a distributed manner (conductors consume and perform work). Conductors are pretty new so if bugs are found please let us know. Jobs have some unique properties that I think make them attractive[3]. The second method here is what is called the W.B.E. (the worker based engine), This is different in that it is a distribution at the engine[4] layer (not at the job layer), this engine allows for running tasks on remote workers (and it can be used in combination with the job concept/layer). Documentation for this can be found @ http://docs.openstack.org/developer/taskflow/workers.html; this WBE is under development, it does work, but it does have a few limitations that still need addressing (see docs link). So that¹s what exists currently (not just in-process things), [1] http://docs.openstack.org/developer/taskflow/jobs.html [2] http://docs.openstack.org/developer/taskflow/conductors.html [3] http://docs.openstack.org/developer/taskflow/jobs.html#features [4] http://docs.openstack.org/developer/taskflow/engines.html -----Original Message----- From: Sandy Walsh <sandy.wa...@rackspace.com> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org> Date: Tuesday, June 17, 2014 at 5:33 AM To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org> Subject: Re: [openstack-dev] [taskflow] Recommendations for the granularity of tasks and their "stickiness" to workers >On 6/17/2014 7:04 AM, Eoghan Glynn wrote: >> Folks, >> >> A question for the taskflow ninjas. >> >> Any thoughts on best practice WRT $subject? >> >> Specifically I have in mind this ceilometer review[1] which adopts >> the approach of using very fine-grained tasks (at the level of an >> individual alarm evaluation) combined with short-term assignments >> to individual workers. >> >> But I'm also thinking of future potential usage of taskflow within >> ceilometer, to support partitioning of work over a scaled-out array >> of central agents. >> >> Does taskflow also naturally support a model whereby more chunky >> tasks (possibly including ongoing periodic work) are assigned to >> workers in a stickier fashion, such that re-balancing of workload >> can easily be triggered when a change is detected in the pool of >> available workers? > >I don't think taskflow today is really focused on load balancing of >tasks. Something like gearman [1] might be better suited in the near term? > >My understanding is that taskflow is really focused on in-process tasks >(with retry, restart, etc) and later will support distributed tasks. But >my data could be stale too. (jharlow?) > >Even still, the decision of smaller tasks vs. chunky ones really comes >down to how much work you want to re-do if there is a failure. I've seen >some uses of taskflow where the breakdown of tasks seemed artificially >small. Meaning, the overhead of going back to the library on an >undo/rewind is greater than the undo itself. > >-S > >[1] http://gearman.org/ > > >_______________________________________________ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev