On Wednesday, August 20, 2014 10:01:55 PM UTC+2, Niphlod wrote: > > well, you're definitely a power-user if you're submitting 1000 > tasks/minute. >
Well, it's more like a burst of 1000 tasks submitted in a minute, with such bursts happening every 10 minutes. The schedule has plenty of time to empty the task list. If you're using a high number of workers you're a perfect candidate for a > new incarnation of the scheduler (yet to come, but it's going to be > "actual" soon) that uses redis as the coordinator instead of the > scheduler_worker table, and relevies the db for a lot of pressure. > Does it mean a redis installation will become a scheduler requirement ? Or will web2py ship with an embedded redis ? I'm not going to put a +1 on the patch because the method you're choosing > is not "atomic" (and I don't see it as the real issue), requiring two > queries that could update the wrong records. > queue_task() DOESN'T use the scheduler_worker table, unless you're using > immediate=True. Queuing 1000 tasks/minute, immediate should be completely > irrelevant. Are you using it? If yes, why ? > > Actually I did use immediate=True, out of misunderstanding its actual behaviour. I rollbacked the patch and moved the queueing code to immediate=False. The deadlock on the scheduler_worker table don't happen anymore. Thanks for pointing this out. > What you should be careful of is IMHO the number of workers and the > heartbeat value. Can I have more details on those figures please ? > > The heartbeat is set to its default 3 seconds. There are 1 main worker, 2 fast workers and 2 slow workers. Most of the tasks are routed to the fast workers. > PS: without proper "pruning" of the scheduler_task table, you'll probably > benefit of indexes here and there to help the db alleviate the relative > pressure of the queries it does. Without figures, though, I can't say more. > I'm indeed interested in knowing which index definitions might help the scheduler. Best regards, Christophe Varoqui OpenSVC -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
