Hi Niphlod,
Thanks for the suggestions! When I DEBUG it, I get this error for my task
function named "run":
DEBUG:web2py.scheduler.ip-172-31-2-63#102815: work to do 8
DEBUG:web2py.scheduler.ip-172-31-2-63#102815: new scheduler_run record
INFO:web2py.scheduler.ip-172-31-2-63#102815:new task 8 "run"
AGIS/default.run
DEBUG:web2py.scheduler.ip-172-31-2-63#102815: new task allocated:
AGIS/default.run
*Traceback (most recent call last):*
* File "/home/www-data/web2py/gluon/shell.py", line 272, in run*
* exec(python_code, _env)*
* File "<string>", line 1, in <module>*
* File "/home/www-data/web2py/gluon/scheduler.py", line 719, in loop*
* self.wrapped_report_task(task, self.async(task))*
* File "/home/www-data/web2py/gluon/scheduler.py", line 342, in async*
* out = multiprocessing.Queue()*
* File "/usr/lib/python2.7/multiprocessing/__init__.py", line 218, in
Queue*
* return Queue(maxsize)*
* File "/usr/lib/python2.7/multiprocessing/queues.py", line 63, in
__init__*
* self._rlock = Lock()*
* File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 147, in
__init__*
* SemLock.__init__(self, SEMAPHORE, 1, 1)*
* File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 75, in
__init__*
* sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)*
*OSError: [Errno 13] Permission denied*
DEBUG:web2py.scheduler.ip-172-31-2-63#102815:........recording heartbeat
(RUNNING)
DEBUG:web2py.scheduler.ip-172-31-2-63#102815:........recording heartbeat
(RUNNING)
DEBUG:web2py.scheduler.ip-172-31-2-63#102815:........recording heartbeat
(RUNNING)
DEBUG:web2py.scheduler.ip-172-31-2-63#102815:........recording heartbeat
(RUNNING)
DEBUG:web2py.scheduler.ip-172-31-2-63#102815: freeing workers that have
not sent heartbeat
DEBUG:web2py.scheduler.ip-172-31-2-63#102815:........recording heartbeat
(RUNNING)
DEBUG:web2py.scheduler.ip-172-31-2-63#102815:........recording heartbeat
(RUNNING)
DEBUG:web2py.scheduler.ip-172-31-2-63#102815:........recording heartbeat
(RUNNING)
DEBUG:web2py.scheduler.ip-172-31-2-63#102815:........recording heartbeat
(RUNNING)
DEBUG:web2py.scheduler.ip-172-31-2-63#102815:........recording heartbeat
(RUNNING)
...
And the worker keeps running all the time.
And I think the permission denied error for multiprocessing is not specific
for my "run" function. Because when I try to run a simple task with name
"testtask"
def testtask():
time.sleep(100)
return 1
it still give the same error:
DEBUG:web2py.scheduler.ip-172-31-2-63#102815: new scheduler_run record
INFO:web2py.scheduler.ip-172-31-2-63#102815:new task 9 "testtask"
AGIS/default.testtask
DEBUG:web2py.scheduler.ip-172-31-2-63#102815: new task allocated:
AGIS/default.testtask
Traceback (most recent call last):
File "/home/www-data/web2py/gluon/shell.py", line 272, in run
exec(python_code, _env)
File "<string>", line 1, in <module>
File "/home/www-data/web2py/gluon/scheduler.py", line 719, in loop
self.wrapped_report_task(task, self.async(task))
File "/home/www-data/web2py/gluon/scheduler.py", line 342, in async
out = multiprocessing.Queue()
File "/usr/lib/python2.7/multiprocessing/__init__.py", line 218, in Queue
return Queue(maxsize)
File "/usr/lib/python2.7/multiprocessing/queues.py", line 63, in __init__
self._rlock = Lock()
File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 147, in
__init__
SemLock.__init__(self, SEMAPHORE, 1, 1)
File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 75, in
__init__
sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 13] Permission denied
Do you know where could be the problem? I've set the owners of all the
files and folders within /web2py/ folder as "www-data".
When I test DEBUG on the server in original region, it is all fine:
EBUG:web2py.scheduler.ip-20-20-10-190#15277:looping...
DEBUG:web2py.scheduler.ip-20-20-10-190#15277: work to do 662
DEBUG:web2py.scheduler.ip-20-20-10-190#15277: new scheduler_run record
INFO:web2py.scheduler.ip-20-20-10-190#15277:new task 662 "testtask"
AGIS/default.testtask
DEBUG:web2py.scheduler.ip-20-20-10-190#15277: new task allocated:
AGIS/default.testtask
DEBUG:web2py.scheduler.ip-20-20-10-190#15277: task starting
DEBUG:web2py.scheduler.ip-20-20-10-190#15277: task started
DEBUG:web2py.scheduler.ip-20-20-10-190#15277:........recording heartbeat
(RUNNING)
DEBUG:web2py.scheduler.ip-20-20-10-190#15277:........recording heartbeat
(RUNNING)
DEBUG:web2py.scheduler.ip-20-20-10-190#15277:........recording heartbeat
(RUNNING)
DEBUG:web2py.scheduler.ip-20-20-10-190#15277: freeing workers that have
not sent heartbeat
DEBUG:web2py.scheduler.ip-20-20-10-190#15277: new task report: COMPLETED
DEBUG:web2py.scheduler.ip-20-20-10-190#15277: result: 1
DEBUG:web2py.scheduler.ip-20-20-10-190#15277: task completed or failed
DEBUG:web2py.scheduler.ip-20-20-10-190#15277: recording task report in db
(COMPLETED)
INFO:web2py.scheduler.ip-20-20-10-190#15277:task completed (COMPLETED)
DEBUG:web2py.scheduler.ip-20-20-10-190#15277:looping...
Thanks very much!
On Thursday, April 30, 2015 at 8:47:48 AM UTC-4, Niphlod wrote:
>
>
>
> On Wednesday, April 29, 2015 at 11:53:56 PM UTC+2, Pengfei Yu wrote:
>>
>> Hi Niphlod and Dave,
>>
>> I think the worker is deleted because the the last heartbeat doesn't
>> change for a while. When I comment out the line "#dead_workers.delete()" in
>> gluon/schdeuler.py, the task is stuck as QUEUED without new run record
>> created. And the worker supposed to be deleted with the last_heartbeat
>> doesn't change. When I keep this line "dead_workers.delete()", the
>> situation remain the same as my original email.
>>
>
> a worker that can't "piggyback" its presence NEEDS to be assumed as dead.
>
>
>>
>> I think the task is never processed, since I added a single test line as
>> "os.system('touch <some_folder>/test.log')" at the beginning of the task
>> function, but this "test.log" file is never created in the server. I guess
>> the worker is died right after it is assigned a task, even before the task
>> is being processed, and then the task is QUEUED again.
>>
>
> in this case, you won't have any scheduler_run records. Those are created
> when the task gets picked up.
>
>
>>
>> I have the configuration in /etc/init/web2py-scheduler,conf, and the
>> service is running, that is why new worker are pop out when previous one
>> dies.
>>
>
> Perfect, but maybe you should monitor if they ALWAYS fail and avoid
> restarting them ?!?!!?
>
>
>>
>> When I set timeout as 10s for the task, it is still the same as before,
>> multiple runs are generated for the same task and they are not ended though
>> the assigned workers are died. I guess for the task, the time from RUNNING
>> to QUEUED again is very short.
>>
>
> This seems to point out that the problem isn't a really long task that
> gets stuck, but rather that your task even before the "10 seconds" has
> issues, or the scheduler has trying to pick that up.
>
>>
>>
>> One difference between the server in original region (works fine) and the
>> current region is that in the current region, I use the port 8000 for the
>> HTTPS access, instead of the default 443. This is because the current one
>> is in China and there are restrictions on HTTP/HTTPS access with port
>> 80/8080/443. I am not sure if this will affect the workers since the worker
>> name only have ip information but not port.
>>
>>
> Shouldn't be a problem.
>
>
>> Niphlod,
>>
>> May I ask how can I enable the DEBUG mode and check the logging? I do see
>> logging in the script of gluon/scheduler.py, but I don't know where I can
>> find the logging information in a log file.
>>
>> Thanks
>>
>
> use logging.conf (enable DEBUG log for root and web2py, then in
> consolehandler) and start the scheduler with
>
> web2py.py -K appname -D 0
>
>
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
