Thank you Hans.. I found celery which works with django. Here is my post. http://groups.google.com/group/celery-users/browse_thread/thread/4d7bed4e381ab964/69f1238fae45944f
First, I will try porting celery to work with pylons and then try with other solutions. Regards, Krish On Sep 12, 5:28 pm, Hans Lellelid <[email protected]> wrote: > [email protected] wrote: > > We need schduler to do certain tasks basically downloading and parsing > > html files & RSS from many other sites. > > > We would like to have a job scheduler with db backend to add and > > remove the tasks using pylons. > > > Have you used any specific job queue with pylons? Could you please > > recommend some? > > I would recommend taking a look at Apache ActiveMQ [1] as a message > broker. It's not written in Python (more on that below), but it's easy > to use and you can interface with it in Python. We use the STOMP > protocol w/ ActiveMQ from Pylons to push messages (usually > JSON-structured) onto worker queues or broadcast messages to topics that > our GUI clients subscribe to (to be notified of changes). > > STOMP protocol has a few client implementations in python. There's > stomp.py [2] which we're using from Pylons right now (though we > simplified it to remove the "listener" parts). There's also stomper [3] > which seems to be better architected, in that the protocol is separate > from the transport. We ended up using Stomper with Twisted for our > worker daemons (which subscribe to the queues), since stomp.py does some > stuff under the hood with spawning off another thread that doesn't > necessarily play nice with other frameworks. > > Anyway, there are other message brokers out there, but ActiveMQ was > *extremely* easy to just get up & running and supported the concept of > topics and queues. In default mode it will persist queues to files, but > can be configured to use a database. There's also MorbidQ (written in > Python, uses Twisted), but it was both difficult to get working and also > seems to hack around with message bodies such that it is not possible to > send binary data, etc. I also don't think it supported persisting the > messages to a database. In short, it seemed to be a somewhat > experimental implementation and didn't feel at all like something I'd > want to trust with anything important. > > I'd be interested in learning the best way to manage socket connections > from within a pylons app. In our case, we're setting up & tearing down > the sockets with every request. This probably isn't optimum, but we > only open those connections somewhat rarely; also I'm honestly not sure > where the *right* place to put that would be. app_globals.py? > (Globals() is a stacked object proxy, right?) > > Anyway, good luck in your search! (And let us know what you end up using.) > > Cheers, > Hans > > [1]http://activemq.apache.org/ > [2]http://code.google.com/p/stomppy/ > [3]http://code.google.com/p/stomper/ --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~----------~----~----~----~------~----~------~--~---
