Hey Chad,
Yea, that is a big issue. I am using flower for monitoring. I only set
that up recently, before that I was also using the logs which I agree is a
huge PITA. Flower works pretty well although if you shut down the web
server, you lose the task history. I'm sure it could be extended to save
the history into a db somewhere if necessary. That level of history hasn't
been necessary for us at this point.
I was already running a Tornado JSON-RPC server to allow access to our API
so it was fairly easy to shove flower in there. I did end up having to
modify some of the flower code to better support prefixes on the URL, but
other than that it works pretty well. Again the only issue is it won't
retain historical data forever, only while it's running.
The reason we run a JSON-RPC server is so our API can either be invoked
client side using our python api which in turn calls the celery api
directly or via json-rpc calls which in turn calls our python api on the
server. I've written a wrapper around Celery and a Tornado JSON-RPC
library that allows a clean abstract interface to our API. To make a
method on a "route" a celery task, a developer only has to use a decorator.
The JSON-RPC server also allows us to make remote calls to our API in
other offices which have their own RabbitMQ brokers setup. So from NY I
can invoke a folder creation call via json-rpc to our LA office which in
turn calls a celery task that gets transported through the LA RabbitMQ
broker and handled by workers running on VM's in LA.
I've also configured celery in such a way that each time a worker picks up
work a fresh python instance is started and sets it's context to the same
project context that the call was originally invoked from. The celery
tasks themselves are very thin wrappers around another API where all of the
actual logic is kept which allows us to very rarely have to restart the
celery workers for API updates. The only time we have to restart the
celery workers/server is if we need to add additional routes or methods to
existing routes. The abstract API interface has also been written so that
we can have different versions available. Each version has it's own queues
setup and when I start a worker up I can specify which versions of the API
and which queues (or routes as I call them in the API) the worker should
handle.
Our interface to the API looks something like (this code is trimmed, it's
only an example):
import psyapifrom psyop.api import filesystem
...
fsr = filesystem.FileSystemRequest()
fsr.add_action("folder", path=project_root_path)
fsr.add_action("create_file", path=pc_path, contents=data, overwrite=True)
fsr.add_action("folder",
path=source_path,
symlink_path=target_path)
fsr.add_action("copy",
source_path=env.get_project_branch_path(),
target_path=new_env.get_project_branch_path(),
ignore_patterns=ignore_patterns)
# get an api instance for version 1
api_instance = psyapi.get_api_version("v1")
# get a json client for version 1 of the api in the la office
json_client = psyapi.get_json_client("v1", host="lam")
# the api_instance and the client instance both have the same interface
api_instance.filesystem.execute_filesystem_request(filesystem_request=fsr.encode("json"))
json_client.filesystem.execute_filesystem_request(filesystem_request=fsr.encode("json"))
FileSystemRequest is a custom class that you can add file system actions to
which will be executed on the server and execute_filesystem_request is a
celery task. When called using the api_instance, it will invoke the celery
task client side whereas when called using json_client, the celery task is
invoked by the json-rpc server. I'm using a RabbitMQ exchange of type
topic, the Celery routing key looks something like psyapi.v1.filesystem.#
and the RabbitMQ queue name that's bound to this key is named something
like psyapi.v1.filesystem.
Technically for a filesystem request you wouldn't have to access the psyapi
module directly. FileSystemRequest uses psyapi internally and has an
execute method that you can pass an argument to telling it to run locally
or on the server:
from psyop.api import filesystem
fsr = filesystem.FileSystemRequest()
fsr.add_action("folder", path=project_root_path)
fsr.execute(local=False)
This is way more information that you were asking for...but once I started
writing a reply I figured I'd just explain kind of how we have everything
setup :). Hopefully it makes some sense!
On Fri, May 30, 2014 at 1:17 PM, Chad Dombrova <[email protected]> wrote:
> Hey tony,
> how do you mange and monitor all of your celery workers? for me that has
> been the main roadblock for celery adoption. we started working on some
> modifications to flower, celery's web-based monitor app, to give it the
> ability to view task logs. without that, it seems like the only option is
> to search the worker's local logs, which seems like a real PITA.
>
> chad.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Python Programming for Autodesk Maya" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/python_inside_maya/CAGq9Q7E5HzohLNeycF2rPmFA8xMWYA%2BqvFLgvV4QwudikC9sGA%40mail.gmail.com
> <https://groups.google.com/d/msgid/python_inside_maya/CAGq9Q7E5HzohLNeycF2rPmFA8xMWYA%2BqvFLgvV4QwudikC9sGA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
--
-tony
--
You received this message because you are subscribed to the Google Groups
"Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/python_inside_maya/CAJhmvsSwqZHzN1Y%2BHY2KHn%3Dzn2EzTVajjzbnh30GyqAUA8Tp%3DQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
