I'm curious what workarounds folks have come up with regarding this issue.
On Monday, February 29, 2016 at 1:41:41 PM UTC-5, Boris Aramis Aguilar Rodríguez wrote: > > Hi, there is an issue driving me crazy with the web2py scheduler: > > If you return something that has a huge size then it will always timeout; > even if the scheduler task correctly finishes. Let me explain with an > example: > > def small_test(): > s = 's'*1256018 > another_s = s > #print s > #print another_s > #print 'FINISHED PROCESS' > return dict(s = s, another_s = another_s, f = 'finished') > > small_test is the function to execute, as you can see a string full of 's' > 1256018 times is. Simple > > So when you enqueue the scheduler every time the output is the same: > http://prnt.sc/a9iarj (screenshot of the TIMEOUT) > > As you can see from the screenshot, the process actually finished; while > logging the scheduler output shows the following: > > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475: work to do 1405 > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475: new scheduler_run > record > INFO:web2py.scheduler.PRTALONENETLAPP-SRV#24475:new task 1405 > "small_test" portal/default.small_test > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475: new task allocated: > portal/default.small_test > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475: task starting > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475: task started > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475: new task report: > COMPLETED > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475: result: {"s": > "ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss$ > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:........recording > heartbeat (RUNNING) > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:........recording > heartbeat (RUNNING) > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:........recording > heartbeat (RUNNING) > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:........recording > heartbeat (RUNNING) > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:........recording > heartbeat (RUNNING) > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475: freeing workers that > have not sent heartbeat > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:........recording > heartbeat (RUNNING) > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:........recording > heartbeat (RUNNING) > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:........recording > heartbeat (RUNNING) > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:........recording > heartbeat (RUNNING) > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:........recording > heartbeat (RUNNING) > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475: freeing workers that > have not sent heartbeat > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:........recording > heartbeat (RUNNING) > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:........recording > heartbeat (RUNNING) > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:........recording > heartbeat (RUNNING) > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:........recording > heartbeat (RUNNING) > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:........recording > heartbeat (RUNNING) > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475: freeing workers that > have not sent heartbeat > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:........recording > heartbeat (RUNNING) > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475: task timeout > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475: recording task report > in db (TIMEOUT) > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475: status TIMEOUT, > stop_time 2016-02-29 11:56:52.393706, run_result {"s": > "sssssssssssssssssssssssssss$ > INFO:web2py.scheduler.PRTALONENETLAPP-SRV#24475:task completed (TIMEOUT) > DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:looping... > INFO:web2py.scheduler.PRTALONENETLAPP-SRV#24475:nothing to do > > > > As you can see there is a TaskReport object in the queue with a COMPLETED > status (I know this because I read the scheduler.py code of web2py) So I'm > pretty sure the task finishes quite fast but then it hangs. > > So I did another test, that doesn't directly use the scheduler but only > calls the executor method from the scheduler and usess process; just like > the scheduler would: > > from gluon.scheduler import Task > from gluon.scheduler import executor > t = Task(app='portal', function='small_test', timeout = 120) > import logging > logging.getLogger().setLevel(logging.DEBUG) > import multiprocessing > queue = multiprocessing.Queue(maxsize = 1) > out = multiprocessing.Queue() > t.task_id = 123 > t.uuid = 'asdfasdf' > p = multiprocessing.Process(target=executor, args=(queue, t, out)) > p.start() > p.join(timeout = 120) > p.is_alive() > > > when the join finishes waiting (2 minutes) if you check for p.is_alive() > it always returns True; but when you do a queue.get() and then instantly > check for p.is_alive() the process finishes!!!!! > > So i noticed the problem is from multiprocessing library, due to the fact > that it can't handle lots of data from a queue (which seems kind of strange > for my case, but I don't know how it is implemented); anyways i found this > bug: http://bugs.python.org/issue8237 and http://bugs.python.org/issue8426 > > The interesting part is it is actually documented (I didn't knew that): > > https://docs.python.org/2/library/multiprocessing.html#multiprocessing-programming > > But in my current implementation this will happen quite often, I'll work > on a work-around but I would really like that web2py scheduler could handle > large data output from my processes for me, but well that is my wish and I > would like to have some guidance on this issue and avoid a work-around. > > Anyway, this should be documented somewhere in web2py too (that probably > could had saved me a week of code reading and debugging); or it should be > managed somehow (I wouldn't naturally expect an output limit besides the > database implementation). > -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
