jokes aside, yeah, you definitely hit scheduler's limits, or, for better saying, limits using a relational database as a queue table. web2py's scheduler can still be optimized, and I feel that 30k tasks are manageable, if they are spread throughout the day (20 tasks a minute if math is not failing me) Managing 15-20 workers is not a problem with a good database backend, 30ish or more is asking for disasters to happen, and that's why redis_scheduler was born. The redis backed one moves to redis the heavy concurrency part, which is the scheduler_worker table and the polling algo for looking for new tasks, but it doesn't move EVERYTHING out of the database. 60-80 workers is probably the limit for redis_backed. Both of them are prolly less than 2k lines of code and they use DAL and redis, but nothing else from the standard library so they're definitely good, but not on par with "specialized" alternatives.
One "grin" of the implementation is that it forks a new process every task, and that's for a basic design principle others do not enforce which is to terminate a long running process going into timeout (and that's because python can't kill a thread, just a process). But that's a problem for people running 40 tasks per second, not 20 a minute. Things to look out for when you reach high numbers like you (i.e. you can see if it helps "staying with the scheduler"): - trim the scheduler_task regularly (having 1m rows in the scheduler_task table, if you do not need them for reporting, can definitely bring down performances) - same deal for scheduler_run (do you need the result, and if you need it, do you need for how much time?!) - create indexes on it-s- the scheduler_task table (one on status, assigned_worker_name helps the workers, on stop_time, next_run_time and status helps the ticker) took that out of the table, you can quite easily jump on a different ship (my advice would be to look at rq before celery, but YMMV) which are HUGE projects. The only problem you may have is that if you use the same backend to store results for your tasks (because they're needed) .... you *may* hit the same limits, i.e. you may have your backend not sized up for the work your apps need to do. -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
