Xianping lin created AIRFLOW-1511:
-------------------------------------
Summary: jobs failed ever in backfill due to concurrency limit
Key: AIRFLOW-1511
URL: https://issues.apache.org/jira/browse/AIRFLOW-1511
Project: Apache Airflow
Issue Type: Bug
Components: backfill
Affects Versions: Airflow 1.8
Reporter: Xianping lin
backfill a DAG which supposed to run daily. some jobs can't start forever.
this is in cluster mode by celery executor.
current time is : Tue Aug 15 04:56:56 PDT 2017.
Those daily job still can't start. see below messages:
*** Log file isn't local.
*** Fetching here:
http://PCDM3:8793/log/user_subs/user_subs_calc/2017-07-08T00:00:00
[2017-08-14 22:11:36,700] {cli.py:391} INFO - Loading pickle id 205
[2017-08-14 22:11:36,818] {base_task_runner.py:112} INFO - Running: ['bash',
'-c', u'airflow run user_subs user_subs_calc 2017-07-08T00:00:00 --pickle 205
--job_id 21317 --raw']
[2017-08-14 22:11:37,723] {base_task_runner.py:95} INFO - Subtask: [2017-08-14
22:11:37,723] {__init__.py:57} INFO - Using executor CeleryExecutor
[2017-08-14 22:11:37,878] {base_task_runner.py:95} INFO - Subtask: [2017-08-14
22:11:37,877] {driver.py:120} INFO - Generating grammar tables from
/usr/lib/python2.7/lib2to3/Grammar.txt
[2017-08-14 22:11:37,911] {base_task_runner.py:95} INFO - Subtask: [2017-08-14
22:11:37,911] {driver.py:120} INFO - Generating grammar tables from
/usr/lib/python2.7/lib2to3/PatternGrammar.txt
[2017-08-14 22:11:38,176] {base_task_runner.py:95} INFO - Subtask: [2017-08-14
22:11:38,176] {cli.py:391} INFO - Loading pickle id 205
[2017-08-14 22:11:38,249] {base_task_runner.py:95} INFO - Subtask: [2017-08-14
22:11:38,249] {models.py:1126} INFO - Dependencies all met for <TaskInstance:
user_subs.user_subs_calc 2017-07-08 00:00:00 [queued]>
[2017-08-14 22:11:38,253] {base_task_runner.py:95} INFO - Subtask: [2017-08-14
22:11:38,253] {models.py:1120} INFO - Dependencies not met for <TaskInstance:
user_subs.user_subs_calc 2017-07-08 00:00:00 [queued]>, dependency 'Task
Instance Slots Available' FAILED: The maximum number of running tasks
(user_subs) for this task's DAG '16' has been reached.
[2017-08-14 22:11:38,262] {base_task_runner.py:95} INFO - Subtask: [2017-08-14
22:11:38,262] {models.py:1300} WARNING -
[2017-08-14 22:11:38,262] {base_task_runner.py:95} INFO - Subtask:
--------------------------------------------------------------------------------
[2017-08-14 22:11:38,262] {base_task_runner.py:95} INFO - Subtask: FIXME:
Rescheduling due to concurrency limits reached at task runtime. Attempt 1 of 4.
State set to NONE.
[2017-08-14 22:11:38,262] {base_task_runner.py:95} INFO - Subtask:
--------------------------------------------------------------------------------
[2017-08-14 22:11:38,262] {base_task_runner.py:95} INFO - Subtask:
[2017-08-14 22:11:38,263] {base_task_runner.py:95} INFO - Subtask: [2017-08-14
22:11:38,262] {models.py:1304} INFO - Queuing into pool None
[2017-08-14 22:11:41,822] {jobs.py:2083} INFO - Task exited with return code 0
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
