Yes, redis can have no response when it's forking to disk, which I found recently, so I stop snapshotting very frequently, and only a snapshot once a day now when the server is not busy. I'd like to take a through check on redis and celery, though my problem had appeared before celery was online.
I had some thoughts like db or cache was very busy so some of requests have to wait, but the number of total requests was not a big number, so many workers are idle while the others are too busy to be killed by harakiri. On Mon, Nov 12, 2012 at 11:53 PM, Jeff Van Voorst <[email protected]>wrote: > I am not familiar with this setup. However, if you haven't check it, I > would use one or more tools to determine if your redis queue has any > problems. For example, if you have lots of requests per second and redis > forks itself and writes to disk, that might disrupt the queue. > > Debugging async methods can require considerable effort and thought. > Personally, I would try to take a systematic approach and verify that all > parts of your system are working as intended (don't just assume that they > are). > > In my opinion, your better options are to find someone familiar with your > setup and to use the celery monitoring tools (if they exist). You can also > use google to search for problems that might occur when using celery with > redis. If I recall correctly, there are some issues under heavy load, but > there are probably work arounds at this point. > > --Jeff > > > On 11/12/12 9:27 AM, Samuel wrote: > > Thanks Jeff, > > I used celery backended by redis, not rabbitmq. > > celery was used like: > > @init_decorator > def view: > > and in this decorator: > > def init_decorator(*args, **kwargs): > celery_task.delay() > > On Mon, Nov 12, 2012 at 11:23 PM, Jeff Van Voorst < > [email protected]> wrote: > >> I don't know if this helps, but when I used celery I had to choose an >> underlying implementation for queues and messaging. Can you monitor the >> celery queue? It might be overrun, be a bottleneck, etc. >> >> Also, it would be helpful to know where you are using celery in the nginx >> + uwsgi stack. There is some documentation about ampq with uwsgi, you >> could also look there if you are using celery in a similar manner (maybe >> via RabbitMQ). >> >> --Jeff >> >> >> >> On 11/12/12 9:18 AM, Samuel wrote: >> >>> I used celery, but I think you didn't mean that. I used processes, not >>> threads. I didn't use the other async mode either. >>> >>> >>> >>> >> _______________________________________________ >> uWSGI mailing list >> [email protected] >> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi >> > > > > -- > *吴焱红(Samuel)* > > 博客: blog.shanbay.com > > > > > > _______________________________________________ > uWSGI mailing > [email protected]http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi > > > > _______________________________________________ > uWSGI mailing list > [email protected] > http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi > > -- *吴焱红(Samuel)* 博客: blog.shanbay.com
_______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
