On Wed, Mar 2, 2011 at 9:42 AM, Dinesh Kapoor <dineshkapoo...@gmail.com> wrote: > 2. Celery is running with concurrency = 2. > 3. I have changed my code so that I call reactor.callfromthread for all > reactor based work. > 4. A lot of times once I schedule the celery task, I get delays ranging from > 3 - 30 seconds when I am running with Celery concurrency =2. > 5. If I reduce the celery concurrency to 1, then everything works great! > 6. Upon debugging it a little bit more, it seems that the selectReactor's > doSelect is where the code is getting stuck on the select().
Celery makes synchronous blocking calls, and uses the multiprocessing library. If I understand you correctly you are attempting to embed the reactor in a celery worker process (your comment on callFromThread seems to say this, at least). The problem is that with a concurrency level of 2 the actual reactor could be running in another process. Some things you can do: - generate webhook tasks instead. Expose your twisted worker code via and twisted.web interface - the celery tasks will simply POST to your twisted.web endpoint and the actual worker code can be twistified. - get celery worker code to make a TCP connection to your twisted server. This is similar to the above idea, except HTTP isn't involved and you have to come up with your own client-server protocol Cheers, Reza -- Reza Lotun mobile: +44 (0)7521 310 763 email: rlo...@gmail.com work: r...@tweetdeck.com twitter: @rlotun _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
