On Sun, Jun 1, 2014 at 8:27 AM, Tony Barbieri <[email protected]> wrote:
> You can define how tasks are routed both by default and on the fly. There > are quite a few options for dealing with this: > http://celery.readthedocs.org/en/latest/userguide/routing.html#id2 > > In the case where you want to direct tasks to the best fitting workers, >> does celery take into account available resources on the workers vs >> requested resources of the task? > > > Not by default, I don't believe it does. This would most likely have to > be written in the routing logic. > > In situations where you want to direct to a subset of workers or a >> specific worker, does that equal a new queue? > > > Yes, I believe it does. A combination of Queues, Exchange types and > routing keys would need to be configured to determine which > workers/consumers should pick up the tasks. > > Do workers have to be preconfigured with "slots" and do tasks consume N >> available slots? > > > Depends. You can let celery decide what kind of concurrency a worker > should have when the worker starts up or you can configure it in the celery > "app" settings. I believe you can also communicate with the consumers > after they have already started and shrink/grow their process pools. > > http://celery.readthedocs.org/en/latest/userguide/workers.html#autoscaling > > In the end I think you would wrap your own setup around celery. I believe > this extra layer would be necessary for some components. Also the way the > exchanges, queues and routes interact would have to be designed based on > all of the various needs. > > Celery is pretty nice to work with once you understand how it all works, > it's flexible and the developer of it is very active. It would be really > interesting to see how far celery could get you writing an application like > this. > > > Right ya, so it sounds like it would purely be based on ultimately having 1 queue per worker in order to have the ability to dispatch a task to what the queue manager (broker) determines to be the best fit for the task (right amount of ram and cpu free). So if the system scales well to, say, more than 1000 queues then it might be feasible. Otherwise being dependent on the RabbitMQ queues as the transport mechanism for a render farm application may not be scalable. Not saying you ever claimed it was, but I was just interested in vetting the concept of it being sustainable as a scalable render farm platform. The broadcast thing sounds interesting though, as you could technically send a message to the entire subgroup of workers, and which ever worker has the matching node id of the message does the actual work. But that sounds like unnecessary chatter anyways. I think that wraps up my question. Celery may be usable as a small render queue for maybe a farm with less than 100 nodes. And it would require writing a bit of routing logic for the broker, and logic for the celery app, in order to support direct worker assignments and accounting the available resources at the broken level for all the workers. > > > On Fri, May 30, 2014 at 6:02 PM, Justin Israel <[email protected]> > wrote: > >> Hey Tony, >> >> I haven't used celery for more than a super simple addition as a local >> background task queue to a Django server, so I thought I would ask >> specifically related to the render farm concept that Marcus mentioned, and >> celery's granularity. Does celery give you the control to define the >> scheduler? In the case where you want to direct tasks to the best fitting >> workers, does celery take into account available resources on the workers >> vs requested resources of the task? In situations where you want to direct >> to a subset of workers or a specific worker, does that equal a new queue? >> Do workers have to be preconfigured with "slots" and do tasks consume N >> available slots? >> >> I can see how well suited celery would be for generalized task >> distribution, but I am curious how much support is included vs how much has >> to be developed as another layer to handle the type of profile of a render >> queue, where the focus is trying to reach and maintain maximum resource >> utilization and task throughput, and the least amount of task re-queues. >> And also with the need for priority sorting. But again maybe that is >> something that has to be a scheduler that sits in front of it actually >> going into the celery queues. >> >> Just curious what your thoughts are on the celery architecture for this >> kind of application, from your experience with it? >> >> >> >> >> On Sat, May 31, 2014 at 6:43 AM, Tony Barbieri <[email protected]> >> wrote: >> >>> Thanks, chad! >>> >>> We haven't used it for that although it's a very interesting idea. >>> We've just started writing some tools for crawling hierarchies, using >>> Celery to distribute the work sounds like a very interesting idea... >>> >>> Also good to know about the persistent option, I'll give that a look if >>> we end up needing it. For now keeping long-term historical data for the >>> types of tasks we're running hasn't been necessary. Typically the >>> historical data is only used for fixing immediate issues that may crop up. >>> >>> >>> On Fri, May 30, 2014 at 2:31 PM, Chad Dombrova <[email protected]> >>> wrote: >>> >>>> cool stuff, tony. do you use celery to do multi-worker distributed >>>> disk crawls, like finding all the files below a given directory? this is >>>> another use case that I'm very interested in. >>>> >>>> also, btw, there is an option in flower to persist the tasks states to >>>> disk, but there's a simple bug that needs to be fixed to get it to work. >>>> >>>> chad. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Python Programming for Autodesk Maya" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/python_inside_maya/CAGq9Q7H35CPhkT%3Dz_n6B4zA0AHS8L9Cw69mf8JDtgf-fTFBZUw%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/python_inside_maya/CAGq9Q7H35CPhkT%3Dz_n6B4zA0AHS8L9Cw69mf8JDtgf-fTFBZUw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> -tony >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Python Programming for Autodesk Maya" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsRthNKfAXDXN3L194c4844shEgOYBisMEaqcBkxhDi%2BwA%40mail.gmail.com >>> <https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsRthNKfAXDXN3L194c4844shEgOYBisMEaqcBkxhDi%2BwA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Python Programming for Autodesk Maya" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA25255sU6%2BMQZZyye-OsgwRXTXd-%3Db3imd6mFOH3jraEQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA25255sU6%2BMQZZyye-OsgwRXTXd-%3Db3imd6mFOH3jraEQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > -tony > > -- > You received this message because you are subscribed to the Google Groups > "Python Programming for Autodesk Maya" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsQjWq3SKfT0SnRHS%2BX8RP1wKaAjTrKGWaT0aamLZk18NQ%40mail.gmail.com > <https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsQjWq3SKfT0SnRHS%2BX8RP1wKaAjTrKGWaT0aamLZk18NQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA3UB5bOzf0RPQzEW-50bDzmin0RHsM1fQM4GCDbOf_94w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
