[
https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17243141#comment-17243141
]
ASF GitHub Bot commented on AIRFLOW-5071:
-----------------------------------------
nathadfield commented on issue #10790:
URL: https://github.com/apache/airflow/issues/10790#issuecomment-737896491
To add some further context, I can consistently replicate this error on
2.0.0b3 on a very simple environment running two Docker containers - webserver
and postgres - on a Python 3.7 image using LocalExecutor and with a
`poke_interval` of `60 * 5`.
```
[2020-12-03 11:52:04,649] {scheduler_job.py:946} INFO - 1 tasks up for
execution:
<TaskInstance: target_dag.wait-task 2020-12-02 00:00:00+00:00
[scheduled]>
[2020-12-03 11:52:04,655] {scheduler_job.py:980} INFO - Figuring out tasks
to run in Pool(name=default_pool) with 128 open slots and 1 task instances
ready to be queued
[2020-12-03 11:52:04,656] {scheduler_job.py:1007} INFO - DAG target_dag has
0/16 running and queued tasks
[2020-12-03 11:52:04,657] {scheduler_job.py:1068} INFO - Setting the
following tasks to queued state:
<TaskInstance: target_dag.wait-task 2020-12-02 00:00:00+00:00
[scheduled]>
[2020-12-03 11:52:04,661] {scheduler_job.py:1110} INFO - Sending
TaskInstanceKey(dag_id='target_dag', task_id='wait-task',
execution_date=datetime.datetime(2020, 12, 2, 0, 0, tzinfo=Timezone('UTC')),
try_number=1) to executor with priority 1 and queue default
[2020-12-03 11:52:04,663] {base_executor.py:79} INFO - Adding to queue:
['airflow', 'tasks', 'run', 'target_dag', 'wait-task',
'2020-12-02T00:00:00+00:00', '--local', '--pool', 'default_pool', '--subdir',
'/usr/local/airflow/dags/target_dag.py']
[2020-12-03 11:52:04,675] {local_executor.py:80} INFO - QueuedLocalWorker
running ['airflow', 'tasks', 'run', 'target_dag', 'wait-task',
'2020-12-02T00:00:00+00:00', '--local', '--pool', 'default_pool', '--subdir',
'/usr/local/airflow/dags/target_dag.py']
[2020-12-03 11:52:04,710] {dagbag.py:440} INFO - Filling up the DagBag from
/usr/local/airflow/dags/target_dag.py
Running <TaskInstance: target_dag.wait-task 2020-12-02T00:00:00+00:00
[queued]> on host 5acdea444946
[2020-12-03 11:52:05 +0000] [568] [INFO] Handling signal: ttin
[2020-12-03 11:52:05 +0000] [11260] [INFO] Booting worker with pid: 11260
[2020-12-03 11:52:05,776] {scheduler_job.py:946} INFO - 1 tasks up for
execution:
<TaskInstance: target_dag.wait-task 2020-12-02 00:00:00+00:00
[scheduled]>
[2020-12-03 11:52:05,783] {scheduler_job.py:980} INFO - Figuring out tasks
to run in Pool(name=default_pool) with 128 open slots and 1 task instances
ready to be queued
[2020-12-03 11:52:05,783] {scheduler_job.py:1007} INFO - DAG target_dag has
0/16 running and queued tasks
[2020-12-03 11:52:05,783] {scheduler_job.py:1068} INFO - Setting the
following tasks to queued state:
<TaskInstance: target_dag.wait-task 2020-12-02 00:00:00+00:00
[scheduled]>
[2020-12-03 11:52:05,791] {scheduler_job.py:1110} INFO - Sending
TaskInstanceKey(dag_id='target_dag', task_id='wait-task',
execution_date=datetime.datetime(2020, 12, 2, 0, 0, tzinfo=Timezone('UTC')),
try_number=1) to executor with priority 1 and queue default
[2020-12-03 11:52:05,791] {base_executor.py:82} ERROR - could not queue task
TaskInstanceKey(dag_id='target_dag', task_id='wait-task',
execution_date=datetime.datetime(2020, 12, 2, 0, 0, tzinfo=Timezone('UTC')),
try_number=1)
[2020-12-03 11:52:05,797] {scheduler_job.py:1208} INFO - Executor reports
execution of target_dag.wait-task execution_date=2020-12-02 00:00:00+00:00
exited with status success for try_number 1
[2020-12-03 11:52:05,808] {scheduler_job.py:1237} ERROR - Executor reports
task instance <TaskInstance: target_dag.wait-task 2020-12-02 00:00:00+00:00
[queued]> finished (success) although the task says its queued. (Info: None)
Was the task killed externally?
```
```
from airflow import models
from airflow.sensors.external_task_sensor import ExternalTaskSensor
from datetime import datetime, timedelta
default_args = {
'owner': 'airflow',
'start_date': datetime(2018, 10, 31),
'retries': 3,
'retry_delay': timedelta(minutes=5)
}
dag_name = 'target_dag'
with models.DAG(dag_name,
default_args=default_args,
schedule_interval='0 0 * * *',
catchup=False,
max_active_runs=5
) as dag:
wait = ExternalTaskSensor(
task_id='wait-task',
external_dag_id='master_dag',
external_task_id='start',
poke_interval=60 * 5,
mode='reschedule'
)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Thousand os Executor reports task instance X finished (success) although the
> task says its queued. Was the task killed externally?
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-5071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5071
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, scheduler
> Affects Versions: 1.10.3
> Reporter: msempere
> Priority: Critical
> Fix For: 1.10.12
>
> Attachments: image-2020-01-27-18-10-29-124.png,
> image-2020-07-08-07-58-42-972.png
>
>
> I'm opening this issue because since I update to 1.10.3 I'm seeing thousands
> of daily messages like the following in the logs:
>
> ```
> {{__init__.py:1580}} ERROR - Executor reports task instance <TaskInstance: X
> 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> {{jobs.py:1484}} ERROR - Executor reports task instance <TaskInstance: X
> 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> ```
> -And looks like this is triggering also thousand of daily emails because the
> flag to send email in case of failure is set to True.-
> I have Airflow setup to use Celery and Redis as a backend queue service.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
