Hey Justin, This is basically what I was hinting at above about granularity. I'm really not sure if Celery would be the right technology for this type of application, at least on it's own. I haven't had to deal with much granularity at this point other than configuring which queues workers will receive work from so I haven't put much thought into using celery for an application like this. Originally I just wanted to point out it's existence to Marcus and see if it could help with what he is trying to do :).
Does celery give you the control to define the scheduler? You can define how tasks are routed both by default and on the fly. There are quite a few options for dealing with this: http://celery.readthedocs.org/en/latest/userguide/routing.html#id2 In the case where you want to direct tasks to the best fitting workers, > does celery take into account available resources on the workers vs > requested resources of the task? Not by default, I don't believe it does. This would most likely have to be written in the routing logic. In situations where you want to direct to a subset of workers or a specific > worker, does that equal a new queue? Yes, I believe it does. A combination of Queues, Exchange types and routing keys would need to be configured to determine which workers/consumers should pick up the tasks. Do workers have to be preconfigured with "slots" and do tasks consume N > available slots? Depends. You can let celery decide what kind of concurrency a worker should have when the worker starts up or you can configure it in the celery "app" settings. I believe you can also communicate with the consumers after they have already started and shrink/grow their process pools. http://celery.readthedocs.org/en/latest/userguide/workers.html#autoscaling In the end I think you would wrap your own setup around celery. I believe this extra layer would be necessary for some components. Also the way the exchanges, queues and routes interact would have to be designed based on all of the various needs. Celery is pretty nice to work with once you understand how it all works, it's flexible and the developer of it is very active. It would be really interesting to see how far celery could get you writing an application like this. On Fri, May 30, 2014 at 6:02 PM, Justin Israel <[email protected]> wrote: > Hey Tony, > > I haven't used celery for more than a super simple addition as a local > background task queue to a Django server, so I thought I would ask > specifically related to the render farm concept that Marcus mentioned, and > celery's granularity. Does celery give you the control to define the > scheduler? In the case where you want to direct tasks to the best fitting > workers, does celery take into account available resources on the workers > vs requested resources of the task? In situations where you want to direct > to a subset of workers or a specific worker, does that equal a new queue? > Do workers have to be preconfigured with "slots" and do tasks consume N > available slots? > > I can see how well suited celery would be for generalized task > distribution, but I am curious how much support is included vs how much has > to be developed as another layer to handle the type of profile of a render > queue, where the focus is trying to reach and maintain maximum resource > utilization and task throughput, and the least amount of task re-queues. > And also with the need for priority sorting. But again maybe that is > something that has to be a scheduler that sits in front of it actually > going into the celery queues. > > Just curious what your thoughts are on the celery architecture for this > kind of application, from your experience with it? > > > > > On Sat, May 31, 2014 at 6:43 AM, Tony Barbieri <[email protected]> wrote: > >> Thanks, chad! >> >> We haven't used it for that although it's a very interesting idea. We've >> just started writing some tools for crawling hierarchies, using Celery to >> distribute the work sounds like a very interesting idea... >> >> Also good to know about the persistent option, I'll give that a look if >> we end up needing it. For now keeping long-term historical data for the >> types of tasks we're running hasn't been necessary. Typically the >> historical data is only used for fixing immediate issues that may crop up. >> >> >> On Fri, May 30, 2014 at 2:31 PM, Chad Dombrova <[email protected]> wrote: >> >>> cool stuff, tony. do you use celery to do multi-worker distributed disk >>> crawls, like finding all the files below a given directory? this is >>> another use case that I'm very interested in. >>> >>> also, btw, there is an option in flower to persist the tasks states to >>> disk, but there's a simple bug that needs to be fixed to get it to work. >>> >>> chad. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Python Programming for Autodesk Maya" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/python_inside_maya/CAGq9Q7H35CPhkT%3Dz_n6B4zA0AHS8L9Cw69mf8JDtgf-fTFBZUw%40mail.gmail.com >>> <https://groups.google.com/d/msgid/python_inside_maya/CAGq9Q7H35CPhkT%3Dz_n6B4zA0AHS8L9Cw69mf8JDtgf-fTFBZUw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> -tony >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Python Programming for Autodesk Maya" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsRthNKfAXDXN3L194c4844shEgOYBisMEaqcBkxhDi%2BwA%40mail.gmail.com >> <https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsRthNKfAXDXN3L194c4844shEgOYBisMEaqcBkxhDi%2BwA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "Python Programming for Autodesk Maya" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA25255sU6%2BMQZZyye-OsgwRXTXd-%3Db3imd6mFOH3jraEQ%40mail.gmail.com > <https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA25255sU6%2BMQZZyye-OsgwRXTXd-%3Db3imd6mFOH3jraEQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- -tony -- You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsQjWq3SKfT0SnRHS%2BX8RP1wKaAjTrKGWaT0aamLZk18NQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
