Hi,
I have a scheduler problem after transferring our web2py server in another
region. We used EC2 for the server and RDS for the mySql database. It works
fine in original region. Right now we move the server and the database to
another region with new AWS EC2 and new RDS. I used the web2py-scheduler
configuration to automatically initiate workers when the server is running.
In the new region, originally, we have 5 workers available. when we created
a new scheduler task waiting to be assigned to the workers, the task is
first been assigned to a worker and a run is created, then quickly this
worker is deleted and the task is queued again, and then the task is
assigned to another active worker and another run is created, then the new
assigned worker is deleted again. It happens all the time if we don't
remove the task, all the previously assigned runs are still running, they
are all for the same task (see attached db_scheduler_run.csv). But the
assigned workers are already deleted and gone from the database
(db_scheduler_worker.csv). Since we start the web2py-scheduler, every time
old workers are deleted, new workers will be generated, so in the table of
"db_scheduler_worker.csv" are the new workers.
I don't understand why the worker is deleted immediately once it has been
assigned to a run for a task.
In the code of gluon/scheduler.py, I found the conditions for deletion of a
worker, but I still cannot find the reason here why my workers get deleted.
self.do_assign_tasks = False if counter % 5 == 0 or mybackedstatus == PICK:
try: # delete dead workers expiration = now - datetime.timedelta( seconds=
self.heartbeat * 3) departure = now - datetime.timedelta(
seconds=self.heartbeat
* 3 * 15) logger.debug( ' freeing workers that have not sent heartbeat')
dead_workers = db( ((sw.last_heartbeat < expiration) & (sw.status ==
ACTIVE)) | ((sw.last_heartbeat < departure) & (sw.status != ACTIVE)) )
dead_workers_name = dead_workers._select(sw.worker_name) db(
(st.assigned_worker_name.belongs(dead_workers_name)) & (st.status ==
RUNNING) ).update(assigned_worker_name='', status=QUEUED)
dead_workers.delete() try: self.is_a_ticker = self.being_a_ticker() except:
logger.error('Error coordinating TICKER') if self.w_stats.status == ACTIVE:
self.do_assign_tasks = True except: logger.error('Error cleaning up')
db.commit()
I think it may related to the heartbeat, but I cannot figure out why. The
server works fine in original region, but this error happens when I copy
the code using git repository from server in original region to server in
new region.
Thanks very much!
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
scheduler_run.id,scheduler_run.task_id,scheduler_run.status,scheduler_run.start_time,scheduler_run.stop_time,scheduler_run.run_output,scheduler_run.run_result,scheduler_run.traceback,scheduler_run.worker_name
474,6,RUNNING,2015-04-29 16:09:55,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#91165
475,6,RUNNING,2015-04-29 16:10:10,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#91169
476,6,RUNNING,2015-04-29 16:10:26,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#91173
477,6,RUNNING,2015-04-29 16:10:42,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#91177
478,6,RUNNING,2015-04-29 16:10:58,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#91181
479,6,RUNNING,2015-04-29 16:30:14,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92122
480,6,RUNNING,2015-04-29 16:30:30,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92126
481,6,RUNNING,2015-04-29 16:30:46,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92130
482,6,RUNNING,2015-04-29 16:31:01,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92134
483,6,RUNNING,2015-04-29 16:31:17,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92138
484,6,RUNNING,2015-04-29 16:31:39,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92173
485,6,RUNNING,2015-04-29 16:31:54,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92177
486,6,RUNNING,2015-04-29 16:32:10,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92181
487,6,RUNNING,2015-04-29 16:32:26,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92185
488,6,RUNNING,2015-04-29 16:32:42,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92189
489,6,RUNNING,2015-04-29 16:33:04,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92233
490,6,RUNNING,2015-04-29 16:33:19,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92237
491,6,RUNNING,2015-04-29 16:33:35,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92241
492,6,RUNNING,2015-04-29 16:33:50,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92245
493,6,RUNNING,2015-04-29 16:34:06,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92249
494,6,RUNNING,2015-04-29 16:34:28,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92303
495,6,RUNNING,2015-04-29 16:34:44,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92307
496,6,RUNNING,2015-04-29 16:34:59,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92311
497,6,RUNNING,2015-04-29 16:35:15,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92315
498,6,RUNNING,2015-04-29 16:35:31,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92320
499,6,RUNNING,2015-04-29 16:35:53,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92363
500,6,RUNNING,2015-04-29 16:36:08,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92367
501,6,RUNNING,2015-04-29 16:36:24,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92371
502,6,RUNNING,2015-04-29 16:36:40,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92375
503,6,RUNNING,2015-04-29 16:36:55,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92379
504,6,RUNNING,2015-04-29 16:37:17,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92421
505,6,RUNNING,2015-04-29 16:37:33,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92425
506,6,RUNNING,2015-04-29 16:37:48,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92429
507,6,RUNNING,2015-04-29 16:38:04,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92433
508,6,RUNNING,2015-04-29 16:38:20,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92437
509,6,RUNNING,2015-04-29 16:38:42,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92467
510,6,RUNNING,2015-04-29 16:38:57,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92471
511,6,RUNNING,2015-04-29 16:39:13,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92475
512,6,RUNNING,2015-04-29 16:39:29,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92479
513,6,RUNNING,2015-04-29 16:39:44,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92483
514,6,RUNNING,2015-04-29 16:40:06,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92511
515,6,RUNNING,2015-04-29 16:40:22,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92515
516,6,RUNNING,2015-04-29 16:40:37,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92519
517,6,RUNNING,2015-04-29 16:40:53,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92523
518,6,RUNNING,2015-04-29 16:41:09,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92527
519,6,RUNNING,2015-04-29 16:41:31,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92555
520,6,RUNNING,2015-04-29 16:41:46,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92560
521,6,RUNNING,2015-04-29 16:42:02,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92564
522,6,RUNNING,2015-04-29 16:42:18,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92568
523,6,RUNNING,2015-04-29 16:42:33,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92572
524,6,RUNNING,2015-04-29 16:42:55,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92601
525,6,RUNNING,2015-04-29 16:43:11,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92605
526,6,RUNNING,2015-04-29 16:43:26,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92609
527,6,RUNNING,2015-04-29 16:43:42,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92613
528,6,RUNNING,2015-04-29 16:43:58,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92617
529,6,RUNNING,2015-04-29 16:44:20,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92645
530,6,RUNNING,2015-04-29 16:44:35,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92649
531,6,RUNNING,2015-04-29 16:44:51,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92653
532,6,RUNNING,2015-04-29 16:45:07,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92657
533,6,RUNNING,2015-04-29 16:45:22,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92661
534,6,RUNNING,2015-04-29 16:45:45,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92691
535,6,RUNNING,2015-04-29 16:46:00,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92695
536,6,RUNNING,2015-04-29 16:46:16,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92699
537,6,RUNNING,2015-04-29 16:46:31,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92703
538,6,RUNNING,2015-04-29 16:46:47,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92707
539,6,RUNNING,2015-04-29 16:47:09,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92735
540,6,RUNNING,2015-04-29 16:47:24,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92739
541,6,RUNNING,2015-04-29 16:47:40,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92743
542,6,RUNNING,2015-04-29 16:47:56,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92747
543,6,RUNNING,2015-04-29 16:48:12,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92751
544,6,RUNNING,2015-04-29 16:48:33,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92780
545,6,RUNNING,2015-04-29 16:48:49,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92784
546,6,RUNNING,2015-04-29 16:49:05,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92788
547,6,RUNNING,2015-04-29 16:49:20,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92792
548,6,RUNNING,2015-04-29 16:49:36,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92797
549,6,RUNNING,2015-04-29 16:49:58,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92825
550,6,RUNNING,2015-04-29 16:50:13,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92829
551,6,RUNNING,2015-04-29 16:50:29,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92833
552,6,RUNNING,2015-04-29 16:50:45,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92837
553,6,RUNNING,2015-04-29 16:51:01,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92841
554,6,RUNNING,2015-04-29 16:51:23,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92869
555,6,RUNNING,2015-04-29 16:51:38,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92873
556,6,RUNNING,2015-04-29 16:51:54,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92877
557,6,RUNNING,2015-04-29 16:52:09,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92882
558,6,RUNNING,2015-04-29 16:52:25,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92886
559,6,RUNNING,2015-04-29 16:52:47,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92915
560,6,RUNNING,2015-04-29 16:53:03,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92919
561,6,RUNNING,2015-04-29 16:53:18,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92923
562,6,RUNNING,2015-04-29 16:53:34,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92927
563,6,RUNNING,2015-04-29 16:53:50,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92931
564,6,RUNNING,2015-04-29 16:54:12,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92959
565,6,RUNNING,2015-04-29 16:54:27,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92963
566,6,RUNNING,2015-04-29 16:54:43,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92967
567,6,RUNNING,2015-04-29 16:54:58,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92971
568,6,RUNNING,2015-04-29 16:55:14,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#92975
569,6,RUNNING,2015-04-29 16:55:36,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#93004
570,6,RUNNING,2015-04-29 16:55:52,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#93009
571,6,RUNNING,2015-04-29 16:56:07,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#93024
572,6,RUNNING,2015-04-29 16:56:23,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#93028
573,6,RUNNING,2015-04-29 16:56:39,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#93032
574,6,RUNNING,2015-04-29 16:57:01,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#93060
575,6,RUNNING,2015-04-29 16:57:16,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#93064
576,6,RUNNING,2015-04-29 16:57:32,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#93068
577,6,RUNNING,2015-04-29 16:57:48,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#93072
578,6,RUNNING,2015-04-29 16:58:03,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#93076
579,6,RUNNING,2015-04-29 16:58:25,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#93105
580,6,RUNNING,2015-04-29 16:58:41,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#93109
581,6,RUNNING,2015-04-29 16:58:56,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#93113
582,6,RUNNING,2015-04-29 16:59:12,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#93117
583,6,RUNNING,2015-04-29 16:59:28,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#93121
584,6,RUNNING,2015-04-29 16:59:50,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#93152
585,6,RUNNING,2015-04-29 17:00:05,<NULL>,<NULL>,<NULL>,<NULL>,ip-172-31-2-63#93156
scheduler_worker.id,scheduler_worker.worker_name,scheduler_worker.first_heartbeat,scheduler_worker.last_heartbeat,scheduler_worker.status,scheduler_worker.is_ticker,scheduler_worker.group_names,scheduler_worker.worker_stats
598,ip-172-31-2-63#93160,2015-04-29 16:59:30,2015-04-29 17:00:18,ACTIVE,True,|main|,"{u'status': u'ACTIVE', u'errors': 0, u'workers': 3, u'queue': 1, u'empty_runs': 1, u'sleep': 3, u'distribution': {u'main': {u'workers': [{u'c': 1, u'name': u'ip-172-31-2-63#93160'}, {u'c': 0, u'name': u'ip-172-31-2-63#93164'}, {u'c': 0, u'name': u'ip-172-31-2-63#93168'}]}}, u'total': 0}"
599,ip-172-31-2-63#93164,2015-04-29 16:59:31,2015-04-29 17:00:28,ACTIVE,False,|main|,"{u'status': u'ACTIVE', u'errors': 0, u'workers': 0, u'queue': 0, u'empty_runs': 20, u'sleep': 3, u'distribution': None, u'total': 0}"
600,ip-172-31-2-63#93168,2015-04-29 16:59:31,2015-04-29 17:00:28,ACTIVE,False,|main|,"{u'status': u'ACTIVE', u'errors': 0, u'workers': 0, u'queue': 0, u'empty_runs': 20, u'sleep': 3, u'distribution': None, u'total': 0}"
