Hi,
I use scheduler from few months with - basically - no big issues. In the
last days I saw a strange behaviour: tasks queued multiple time when in my
shceduler.py I have only one scheduler.queue_task instruction. At first I
thougth of multiple scheduler instances thrown by cron jobs (I even don't
know if this is possible), but I checked and I have only one scheduler
instance running.
So I tested this:
db.py:
db =
DAL('postgres://user:[email protected]/oadb',pool_size=1,check_reserved=['all'],
migrate=True)
scheduler.py:
db.define_table('testtable',Field('field1','integer'),migrate=True)
def test_insert(tablename,value):
db[tablename].insert(field1=value)
db.commit()
return True
from gluon.scheduler import Scheduler
scheduler = Scheduler(db)
scheduler.queue_task('test_insert',
pargs=["testtable",1],
task_name='insert test',
group_name='main',
application_name='scrapers'
)
When I run from the shell prompt this:
/usr/local/bin/python $HOME/webapps/oawa/web2py/web2py.py
--password='<recycle>' --scheduler=scrapers -D 0
I got this log:
web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2013
Version 2.4.7-stable+timestamp.2013.05.24.17.48.47
Database drivers available: SQLite(sqlite3), MySQL(pymysql),
MySQL(MySQLdb), PostgreSQL(psycopg2), PostgreSQL(pg8000), MSSQL(pyodbc),
DB2(pyodbc), Teradata(pyodbc), Ingres(pyodbc), IMAP(imaplib)
starting single-scheduler for "scrapers"...
DEBUG:web2py.dal:Your database version does not support the JSON data type
(using TEXT instead)
DEBUG:web2py.scheduler.web394.webfaction.com#385289:defining tables
(migrate=True)
DEBUG:web2py.scheduler.web394.webfaction.com#385289:thread building own DAL
object
DEBUG:web2py.scheduler.web394.webfaction.com#385289:looping...
DEBUG:web2py.dal:Your database version does not support the JSON data type
(using TEXT instead)
DEBUG:web2py.scheduler.web394.webfaction.com#385289:defining tables
(migrate=False)
DEBUG:web2py.scheduler.web394.webfaction.com#385289: freeing workers
that have not sent heartbeat
INFO:web2py.scheduler.web394.webfaction.com#385289:TICKER: I'm a ticker
INFO:web2py.scheduler.web394.webfaction.com#385289:nothing to do
DEBUG:web2py.scheduler.web394.webfaction.com#385289:sleeping...
DEBUG:web2py.scheduler.web394.webfaction.com#385289:........recording
heartbeat (ACTIVE)
DEBUG:web2py.scheduler.web394.webfaction.com#385289:looping...
INFO:web2py.scheduler.web394.webfaction.com#385289:nothing to do
DEBUG:web2py.scheduler.web394.webfaction.com#385289:sleeping...
DEBUG:web2py.scheduler.web394.webfaction.com#385289:........recording
heartbeat (ACTIVE)
DEBUG:web2py.scheduler.web394.webfaction.com#385289:looping...
INFO:web2py.scheduler.web394.webfaction.com#385289:nothing to do
DEBUG:web2py.scheduler.web394.webfaction.com#385289:sleeping...
DEBUG:web2py.scheduler.web394.webfaction.com#385289:........recording
heartbeat (ACTIVE)
DEBUG:web2py.scheduler.web394.webfaction.com#385289:looping...
INFO:web2py.scheduler.web394.webfaction.com#385289:nothing to do
DEBUG:web2py.scheduler.web394.webfaction.com#385289:sleeping...
DEBUG:web2py.scheduler.web394.webfaction.com#385289:........recording
heartbeat (ACTIVE)
DEBUG:web2py.scheduler.web394.webfaction.com#385289:looping...
INFO:web2py.scheduler.web394.webfaction.com#385289:nothing to do
DEBUG:web2py.scheduler.web394.webfaction.com#385289:sleeping...
DEBUG:web2py.scheduler.web394.webfaction.com#385289:........recording
heartbeat (ACTIVE)
DEBUG:web2py.scheduler.web394.webfaction.com#385289: freeing workers
that have not sent heartbeat
INFO:web2py.scheduler.web394.webfaction.com#385289:TICKER: I'm a ticker
DEBUG:web2py.scheduler.web394.webfaction.com#385289:looping...
DEBUG:web2py.scheduler.web394.webfaction.com#385289:Assigning tasks...
INFO:web2py.scheduler.web394.webfaction.com#385289:TICKER: workers are 1
INFO:web2py.scheduler.web394.webfaction.com#385289:TICKER: tasks are 50
DEBUG:web2py.scheduler.web394.webfaction.com#385289:Tasks assigned...
DEBUG:web2py.scheduler.web394.webfaction.com#385289:sleeping...
DEBUG:web2py.scheduler.web394.webfaction.com#385289:........recording
heartbeat (ACTIVE)
DEBUG:web2py.scheduler.web394.webfaction.com#385289:looping...
DEBUG:web2py.scheduler.web394.webfaction.com#385289: work to do 165100
DEBUG:web2py.scheduler.web394.webfaction.com#385289: new scheduler_run
record
INFO:web2py.scheduler.web394.webfaction.com#385289:new task 165100 "insert
test" scrapers.test_insert
DEBUG:web2py.scheduler.web394.webfaction.com#385289: new task allocated:
scrapers.test_insert
DEBUG:web2py.scheduler.web394.webfaction.com#385289: task starting
DEBUG:web2py.scheduler.web394.webfaction.com#385289: task started
DEBUG:web2py.scheduler.web394.webfaction.com#385289: new task report:
COMPLETED
DEBUG:web2py.scheduler.web394.webfaction.com#385289: result: true
DEBUG:web2py.scheduler.web394.webfaction.com#385289: task completed or
failed
DEBUG:web2py.scheduler.web394.webfaction.com#385289: recording task report
in db (COMPLETED)
INFO:web2py.scheduler.web394.webfaction.com#385289:task completed
(COMPLETED)
DEBUG:web2py.scheduler.web394.webfaction.com#385289:looping...
DEBUG:web2py.scheduler.web394.webfaction.com#385289: work to do 165110
DEBUG:web2py.scheduler.web394.webfaction.com#385289: new scheduler_run
record
INFO:web2py.scheduler.web394.webfaction.com#385289:new task 165110 "insert
test" scrapers.test_insert
DEBUG:web2py.scheduler.web394.webfaction.com#385289: new task allocated:
scrapers.test_insert
DEBUG:web2py.scheduler.web394.webfaction.com#385289: task starting
DEBUG:web2py.scheduler.web394.webfaction.com#385289: task started
DEBUG:web2py.scheduler.web394.webfaction.com#385289: new task report:
COMPLETED
DEBUG:web2py.scheduler.web394.webfaction.com#385289: result: true
DEBUG:web2py.scheduler.web394.webfaction.com#385289: task completed or
failed
DEBUG:web2py.scheduler.web394.webfaction.com#385289: recording task report
in db (COMPLETED)
INFO:web2py.scheduler.web394.webfaction.com#385289:task completed
(COMPLETED)
DEBUG:web2py.scheduler.web394.webfaction.com#385289:looping...
DEBUG:web2py.scheduler.web394.webfaction.com#385289: work to do 165113
DEBUG:web2py.scheduler.web394.webfaction.com#385289: new scheduler_run
record
INFO:web2py.scheduler.web394.webfaction.com#385289:new task 165113 "insert
test" scrapers.test_insert
DEBUG:web2py.scheduler.web394.webfaction.com#385289: new task allocated:
scrapers.test_insert
DEBUG:web2py.scheduler.web394.webfaction.com#385289: task starting
DEBUG:web2py.scheduler.web394.webfaction.com#385289: task started
DEBUG:web2py.scheduler.web394.webfaction.com#385289: new task report:
COMPLETED
DEBUG:web2py.scheduler.web394.webfaction.com#385289: result: true
DEBUG:web2py.scheduler.web394.webfaction.com#385289: task completed or
failed
DEBUG:web2py.scheduler.web394.webfaction.com#385289: recording task report
in db (COMPLETED)
INFO:web2py.scheduler.web394.webfaction.com#385289:task completed
(COMPLETED)
DEBUG:web2py.scheduler.web394.webfaction.com#385289:looping...
DEBUG:web2py.scheduler.web394.webfaction.com#385289: work to do 165109
DEBUG:web2py.scheduler.web394.webfaction.com#385289: new scheduler_run
record
INFO:web2py.scheduler.web394.webfaction.com#385289:new task 165109 "insert
test" scrapers.test_insert
DEBUG:web2py.scheduler.web394.webfaction.com#385289: new task allocated:
scrapers.test_insert
DEBUG:web2py.scheduler.web394.webfaction.com#385289: task starting
DEBUG:web2py.scheduler.web394.webfaction.com#385289: task started
DEBUG:web2py.scheduler.web394.webfaction.com#385289: new task report:
COMPLETED
DEBUG:web2py.scheduler.web394.webfaction.com#385289: result: true
DEBUG:web2py.scheduler.web394.webfaction.com#385289: task completed or
failed
DEBUG:web2py.scheduler.web394.webfaction.com#385289: recording task report
in db (COMPLETED)
INFO:web2py.scheduler.web394.webfaction.com#385289:task completed
(COMPLETED)
DEBUG:web2py.scheduler.web394.webfaction.com#385289:looping...
DEBUG:web2py.scheduler.web394.webfaction.com#385289: work to do 165114
DEBUG:web2py.scheduler.web394.webfaction.com#385289: new scheduler_run
record
INFO:web2py.scheduler.web394.webfaction.com#385289:new task 165114 "insert
test" scrapers.test_insert
DEBUG:web2py.scheduler.web394.webfaction.com#385289: new task allocated:
scrapers.test_insert
DEBUG:web2py.scheduler.web394.webfaction.com#385289: task starting
DEBUG:web2py.scheduler.web394.webfaction.com#385289: task started
DEBUG:web2py.scheduler.web394.webfaction.com#385289: new task report:
COMPLETED
DEBUG:web2py.scheduler.web394.webfaction.com#385289: result: true
DEBUG:web2py.scheduler.web394.webfaction.com#385289: task completed or
failed
DEBUG:web2py.scheduler.web394.webfaction.com#385289: recording task report
in db (COMPLETED)
INFO:web2py.scheduler.web394.webfaction.com#385289:task completed
(COMPLETED)
DEBUG:web2py.scheduler.web394.webfaction.com#385289:looping...
DEBUG:web2py.scheduler.web394.webfaction.com#385289: work to do 165112
DEBUG:web2py.scheduler.web394.webfaction.com#385289: new scheduler_run
record
INFO:web2py.scheduler.web394.webfaction.com#385289:new task 165112 "insert
test" scrapers.test_insert
DEBUG:web2py.scheduler.web394.webfaction.com#385289: new task allocated:
scrapers.test_insert
DEBUG:web2py.scheduler.web394.webfaction.com#385289: task starting
DEBUG:web2py.scheduler.web394.webfaction.com#385289: task started
DEBUG:web2py.scheduler.web394.webfaction.com#385289: new task report:
COMPLETED
DEBUG:web2py.scheduler.web394.webfaction.com#385289: result: true
DEBUG:web2py.scheduler.web394.webfaction.com#385289: task completed or
failed
DEBUG:web2py.scheduler.web394.webfaction.com#385289: recording task report
in db (COMPLETED)
INFO:web2py.scheduler.web394.webfaction.com#385289:task completed
(COMPLETED)
DEBUG:web2py.scheduler.web394.webfaction.com#385289:looping...
DEBUG:web2py.scheduler.web394.webfaction.com#385289: work to do 165107
DEBUG:web2py.scheduler.web394.webfaction.com#385289: new scheduler_run
record
...
keeps logging this way thousands of time, filling scheduler_task table with
more the 160.000 rows.
This does not happen if I comment db.commit() line, in this case there is
(of course) no data insertion in testtable, the task is completed but there
is only 1 task in scheduler_task table.
My setup:
2.4.7-stable+timestamp.2013.05.24.17.48.47
(Running on Apache/2.2.17 (Unix) mod_wsgi/3.3 Python/2.7.5)
PostgreSQL 9.1.9
hosting: webfaction.com
I really don't know what it's not working properly.
--
---
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
