[openstack-dev] [taskflow] Recommendations for the granularity of tasks and their stickiness to workers

2014-06-17 Thread Eoghan Glynn
Folks, A question for the taskflow ninjas. Any thoughts on best practice WRT $subject? Specifically I have in mind this ceilometer review[1] which adopts the approach of using very fine-grained tasks (at the level of an individual alarm evaluation) combined with short-term assignments to

Re: [openstack-dev] [taskflow] Recommendations for the granularity of tasks and their stickiness to workers

2014-06-17 Thread Julien Danjou
On Tue, Jun 17 2014, Eoghan Glynn wrote: Any thoughts on best practice WRT $subject? First thing on my mind is that having smaller task can allow to have a better repartition of the work load. :) -- Julien Danjou # Free Software hacker # http://julien.danjou.info signature.asc Description:

Re: [openstack-dev] [taskflow] Recommendations for the granularity of tasks and their stickiness to workers

2014-06-17 Thread Sandy Walsh
On 6/17/2014 7:04 AM, Eoghan Glynn wrote: Folks, A question for the taskflow ninjas. Any thoughts on best practice WRT $subject? Specifically I have in mind this ceilometer review[1] which adopts the approach of using very fine-grained tasks (at the level of an individual alarm

Re: [openstack-dev] [taskflow] Recommendations for the granularity of tasks and their stickiness to workers

2014-06-17 Thread Joshua Harlow
Howdy, Sandy is correct, we aren't doing automatic load-balancing of tasks/jobs (currently). Though there is nothing stopping this from being implemented (and I of course would recommend adding it to taskflow rather than jumping to gearman, but I am obviously biased), A feature that appeared

Re: [openstack-dev] [taskflow] Recommendations for the granularity of tasks and their stickiness to workers

2014-06-17 Thread Joshua Harlow
Back on the distributed subject (since this deserves a different email), In the newest taskflow release (0.3.x) we have 2 mechanisms for distribution outside of a process. One is the job/jobboard[1] conductor[2] concepts, These concepts allow for atomic ownership of a 'job' and conductors act

Re: [openstack-dev] [taskflow] Recommendations for the granularity of tasks and their stickiness to workers

2014-06-17 Thread Eoghan Glynn
Thanks to Joshua and Sandy for the clarifications on the finer points of taskflow features and usage. I'll consume the docco linked, and will be back to annoy ye with further questions :) Cheers, Eoghan Back on the distributed subject (since this deserves a different email), In the newest

Re: [openstack-dev] [taskflow] Recommendations for the granularity of tasks and their stickiness to workers

2014-06-17 Thread Joshua Harlow
Cool, feel free to jump on IRC[1] if u need in-person details/questions or other (consultation is free, haha). [1] irc://chat.freenode.net/openstack-state-management -Josh -Original Message- From: Eoghan Glynn egl...@redhat.com Reply-To: OpenStack Development Mailing List (not for usage