cbuffett commented on issue #34497:
URL: https://github.com/apache/airflow/issues/34497#issuecomment-1736042554
I'm on Airflow 2.6.1.
I ran your test script, but it had some errors due to value being passed to
`execution_date_fn` not being a function. After fixing that (see below), I was
able to reproduce the issue
```
import datetime
from airflow import DAG
from airflow.sensors.external_task import ExternalTaskSensor
from airflow.models.taskinstance import State
def return_date(dt):
return dt
default_args = {
'owner': 'test',
'start_date': datetime.datetime(2023, 1, 1)
}
dag = DAG(dag_id='DAG-1',
default_args=default_args,
schedule='@once',
catchup=False
)
with dag:
external_task_sensor = ExternalTaskSensor(
task_id='external_dag_sensor',
poke_interval=60,
timeout=300,
soft_fail=True,
retries=0,
external_dag_id="some_dag",
execution_date_fn=return_date,
allowed_states=[State.SUCCESS],
failed_states=[State.FAILED],
mode="reschedule"
)
if __name__ == "__main__" :
dag.test()
```
```
[2023-09-26T11:01:48.848-0700] {dag.py:3716} INFO - dagrun id: DAG-1
[2023-09-26T11:01:48.859-0700] {dag.py:3733} INFO - created dagrun <DagRun
DAG-1 @ 2023-09-26T18:01:48.838825+00:00:
manual__2023-09-26T18:01:48.838825+00:00, state:running, queued_at: None.
externally triggered: False>
C:\Users\cbuffett\AppData\Roaming\Python\Python38\site-packages\airflow\models\dag.py:3725
RemovedInAirflow3Warning: Calling `DAG.create_dagrun()` without an explicit
data interval is deprecated
[2023-09-26T11:01:48.866-0700] {dag.py:3683} INFO -
*****************************************************
[2023-09-26T11:01:48.866-0700] {dag.py:3687} INFO - Running task
external_dag_sensor
[2023-09-26 11:01:48,921] {taskinstance.py:1545} INFO - Exporting env vars:
AIRFLOW_CTX_DAG_OWNER='test' AIRFLOW_CTX_DAG_ID='DAG-1'
AIRFLOW_CTX_TASK_ID='external_dag_sensor'
AIRFLOW_CTX_EXECUTION_DATE='2023-09-26T18:01:48.838825+00:00'
AIRFLOW_CTX_TRY_NUMBER='1'
AIRFLOW_CTX_DAG_RUN_ID='manual__2023-09-26T18:01:48.838825+00:00'
[2023-09-26T11:01:48.921-0700] {taskinstance.py:1545} INFO - Exporting env
vars: AIRFLOW_CTX_DAG_OWNER='test' AIRFLOW_CTX_DAG_ID='DAG-1'
AIRFLOW_CTX_TASK_ID='external_dag_sensor'
AIRFLOW_CTX_EXECUTION_DATE='2023-09-26T18:01:48.838825+00:00'
AIRFLOW_CTX_TRY_NUMBER='1'
AIRFLOW_CTX_DAG_RUN_ID='manual__2023-09-26T18:01:48.838825+00:00'
[2023-09-26 11:01:48,926] {external_task.py:247} INFO - Poking for DAG
'some_dag' on 2023-09-26T18:01:48.838825+00:00 ...
[2023-09-26T11:01:48.926-0700] {external_task.py:247} INFO - Poking for DAG
'some_dag' on 2023-09-26T18:01:48.838825+00:00 ...
[2023-09-26 11:01:48,962] {taskinstance.py:1784} INFO - Rescheduling task,
marking task as UP_FOR_RESCHEDULE
[2023-09-26T11:01:48.962-0700] {taskinstance.py:1784} INFO - Rescheduling
task, marking task as UP_FOR_RESCHEDULE
[2023-09-26T11:01:48.963-0700] {dag.py:3691} INFO - external_dag_sensor ran
successfully!
[2023-09-26T11:01:48.963-0700] {dag.py:3694} INFO -
*****************************************************
[2023-09-26T11:02:48.937-0700] {dag.py:3683} INFO -
*****************************************************
[2023-09-26T11:02:48.938-0700] {dag.py:3687} INFO - Running task
external_dag_sensor
[2023-09-26 11:02:48,989] {taskinstance.py:1545} INFO - Exporting env vars:
AIRFLOW_CTX_DAG_OWNER='test' AIRFLOW_CTX_DAG_ID='DAG-1'
AIRFLOW_CTX_TASK_ID='external_dag_sensor'
AIRFLOW_CTX_EXECUTION_DATE='2023-09-26T18:01:48.838825+00:00'
AIRFLOW_CTX_DAG_RUN_ID='manual__2023-09-26T18:01:48.838825+00:00'
[2023-09-26T11:02:48.989-0700] {taskinstance.py:1545} INFO - Exporting env
vars: AIRFLOW_CTX_DAG_OWNER='test' AIRFLOW_CTX_DAG_ID='DAG-1'
AIRFLOW_CTX_TASK_ID='external_dag_sensor'
AIRFLOW_CTX_EXECUTION_DATE='2023-09-26T18:01:48.838825+00:00'
AIRFLOW_CTX_DAG_RUN_ID='manual__2023-09-26T18:01:48.838825+00:00'
[2023-09-26 11:02:48,995] {external_task.py:247} INFO - Poking for DAG
'some_dag' on 2023-09-26T18:01:48.838825+00:00 ...
[2023-09-26T11:02:48.995-0700] {external_task.py:247} INFO - Poking for DAG
'some_dag' on 2023-09-26T18:01:48.838825+00:00 ...
[2023-09-26 11:02:49,022] {taskinstance.py:1784} INFO - Rescheduling task,
marking task as UP_FOR_RESCHEDULE
[2023-09-26T11:02:49.022-0700] {taskinstance.py:1784} INFO - Rescheduling
task, marking task as UP_FOR_RESCHEDULE
[2023-09-26T11:02:49.023-0700] {dag.py:3691} INFO - external_dag_sensor ran
successfully!
[2023-09-26T11:02:49.023-0700] {dag.py:3694} INFO -
*****************************************************
[2023-09-26T11:03:49.042-0700] {dag.py:3683} INFO -
*****************************************************
[2023-09-26T11:03:49.042-0700] {dag.py:3687} INFO - Running task
external_dag_sensor
[2023-09-26 11:03:49,106] {taskinstance.py:1545} INFO - Exporting env vars:
AIRFLOW_CTX_DAG_OWNER='test' AIRFLOW_CTX_DAG_ID='DAG-1'
AIRFLOW_CTX_TASK_ID='external_dag_sensor'
AIRFLOW_CTX_EXECUTION_DATE='2023-09-26T18:01:48.838825+00:00'
AIRFLOW_CTX_TRY_NUMBER='-1'
AIRFLOW_CTX_DAG_RUN_ID='manual__2023-09-26T18:01:48.838825+00:00'
[2023-09-26T11:03:49.106-0700] {taskinstance.py:1545} INFO - Exporting env
vars: AIRFLOW_CTX_DAG_OWNER='test' AIRFLOW_CTX_DAG_ID='DAG-1'
AIRFLOW_CTX_TASK_ID='external_dag_sensor'
AIRFLOW_CTX_EXECUTION_DATE='2023-09-26T18:01:48.838825+00:00'
AIRFLOW_CTX_TRY_NUMBER='-1'
AIRFLOW_CTX_DAG_RUN_ID='manual__2023-09-26T18:01:48.838825+00:00'
[2023-09-26 11:03:49,112] {external_task.py:247} INFO - Poking for DAG
'some_dag' on 2023-09-26T18:01:48.838825+00:00 ...
[2023-09-26T11:03:49.112-0700] {external_task.py:247} INFO - Poking for DAG
'some_dag' on 2023-09-26T18:01:48.838825+00:00 ...
[2023-09-26 11:03:49,145] {taskinstance.py:1784} INFO - Rescheduling task,
marking task as UP_FOR_RESCHEDULE
[2023-09-26T11:03:49.145-0700] {taskinstance.py:1784} INFO - Rescheduling
task, marking task as UP_FOR_RESCHEDULE
[2023-09-26T11:03:49.145-0700] {dag.py:3691} INFO - external_dag_sensor ran
successfully!
[2023-09-26T11:03:49.145-0700] {dag.py:3694} INFO -
*****************************************************
[2023-09-26T11:04:49.123-0700] {dag.py:3683} INFO -
*****************************************************
[2023-09-26T11:04:49.124-0700] {dag.py:3687} INFO - Running task
external_dag_sensor
[2023-09-26 11:04:49,182] {taskinstance.py:1545} INFO - Exporting env vars:
AIRFLOW_CTX_DAG_OWNER='test' AIRFLOW_CTX_DAG_ID='DAG-1'
AIRFLOW_CTX_TASK_ID='external_dag_sensor'
AIRFLOW_CTX_EXECUTION_DATE='2023-09-26T18:01:48.838825+00:00'
AIRFLOW_CTX_TRY_NUMBER='-2'
AIRFLOW_CTX_DAG_RUN_ID='manual__2023-09-26T18:01:48.838825+00:00'
[2023-09-26T11:04:49.182-0700] {taskinstance.py:1545} INFO - Exporting env
vars: AIRFLOW_CTX_DAG_OWNER='test' AIRFLOW_CTX_DAG_ID='DAG-1'
AIRFLOW_CTX_TASK_ID='external_dag_sensor'
AIRFLOW_CTX_EXECUTION_DATE='2023-09-26T18:01:48.838825+00:00'
AIRFLOW_CTX_TRY_NUMBER='-2'
AIRFLOW_CTX_DAG_RUN_ID='manual__2023-09-26T18:01:48.838825+00:00'
[2023-09-26 11:04:49,188] {external_task.py:247} INFO - Poking for DAG
'some_dag' on 2023-09-26T18:01:48.838825+00:00 ...
[2023-09-26T11:04:49.188-0700] {external_task.py:247} INFO - Poking for DAG
'some_dag' on 2023-09-26T18:01:48.838825+00:00 ...
[2023-09-26 11:04:49,217] {taskinstance.py:1784} INFO - Rescheduling task,
marking task as UP_FOR_RESCHEDULE
[2023-09-26T11:04:49.217-0700] {taskinstance.py:1784} INFO - Rescheduling
task, marking task as UP_FOR_RESCHEDULE
[2023-09-26T11:04:49.218-0700] {dag.py:3691} INFO - external_dag_sensor ran
successfully!
[2023-09-26T11:04:49.218-0700] {dag.py:3694} INFO -
*****************************************************
[2023-09-26T11:05:49.200-0700] {dag.py:3683} INFO -
*****************************************************
[2023-09-26T11:05:49.200-0700] {dag.py:3687} INFO - Running task
external_dag_sensor
[2023-09-26 11:05:49,269] {taskinstance.py:1545} INFO - Exporting env vars:
AIRFLOW_CTX_DAG_OWNER='test' AIRFLOW_CTX_DAG_ID='DAG-1'
AIRFLOW_CTX_TASK_ID='external_dag_sensor'
AIRFLOW_CTX_EXECUTION_DATE='2023-09-26T18:01:48.838825+00:00'
AIRFLOW_CTX_TRY_NUMBER='-3'
AIRFLOW_CTX_DAG_RUN_ID='manual__2023-09-26T18:01:48.838825+00:00'
[2023-09-26T11:05:49.269-0700] {taskinstance.py:1545} INFO - Exporting env
vars: AIRFLOW_CTX_DAG_OWNER='test' AIRFLOW_CTX_DAG_ID='DAG-1'
AIRFLOW_CTX_TASK_ID='external_dag_sensor'
AIRFLOW_CTX_EXECUTION_DATE='2023-09-26T18:01:48.838825+00:00'
AIRFLOW_CTX_TRY_NUMBER='-3'
AIRFLOW_CTX_DAG_RUN_ID='manual__2023-09-26T18:01:48.838825+00:00'
[2023-09-26 11:05:49,274] {external_task.py:247} INFO - Poking for DAG
'some_dag' on 2023-09-26T18:01:48.838825+00:00 ...
[2023-09-26T11:05:49.274-0700] {external_task.py:247} INFO - Poking for DAG
'some_dag' on 2023-09-26T18:01:48.838825+00:00 ...
[2023-09-26 11:05:49,307] {taskinstance.py:1784} INFO - Rescheduling task,
marking task as UP_FOR_RESCHEDULE
[2023-09-26T11:05:49.307-0700] {taskinstance.py:1784} INFO - Rescheduling
task, marking task as UP_FOR_RESCHEDULE
[2023-09-26T11:05:49.308-0700] {dag.py:3691} INFO - external_dag_sensor ran
successfully!
[2023-09-26T11:05:49.309-0700] {dag.py:3694} INFO -
*****************************************************
[2023-09-26T11:06:49.287-0700] {dag.py:3683} INFO -
*****************************************************
[2023-09-26T11:06:49.288-0700] {dag.py:3687} INFO - Running task
external_dag_sensor
[2023-09-26 11:06:49,346] {taskinstance.py:1545} INFO - Exporting env vars:
AIRFLOW_CTX_DAG_OWNER='test' AIRFLOW_CTX_DAG_ID='DAG-1'
AIRFLOW_CTX_TASK_ID='external_dag_sensor'
AIRFLOW_CTX_EXECUTION_DATE='2023-09-26T18:01:48.838825+00:00'
AIRFLOW_CTX_TRY_NUMBER='-4'
AIRFLOW_CTX_DAG_RUN_ID='manual__2023-09-26T18:01:48.838825+00:00'
[2023-09-26T11:06:49.346-0700] {taskinstance.py:1545} INFO - Exporting env
vars: AIRFLOW_CTX_DAG_OWNER='test' AIRFLOW_CTX_DAG_ID='DAG-1'
AIRFLOW_CTX_TASK_ID='external_dag_sensor'
AIRFLOW_CTX_EXECUTION_DATE='2023-09-26T18:01:48.838825+00:00'
AIRFLOW_CTX_TRY_NUMBER='-4'
AIRFLOW_CTX_DAG_RUN_ID='manual__2023-09-26T18:01:48.838825+00:00'
[2023-09-26 11:06:49,353] {external_task.py:247} INFO - Poking for DAG
'some_dag' on 2023-09-26T18:01:48.838825+00:00 ...
[2023-09-26T11:06:49.353-0700] {external_task.py:247} INFO - Poking for DAG
'some_dag' on 2023-09-26T18:01:48.838825+00:00 ...
[2023-09-26 11:06:49,383] {taskinstance.py:1784} INFO - Rescheduling task,
marking task as UP_FOR_RESCHEDULE
[2023-09-26T11:06:49.383-0700] {taskinstance.py:1784} INFO - Rescheduling
task, marking task as UP_FOR_RESCHEDULE
[2023-09-26T11:06:49.383-0700] {dag.py:3691} INFO - external_dag_sensor ran
successfully!
[2023-09-26T11:06:49.384-0700] {dag.py:3694} INFO -
*****************************************************
!!! SHOULD HAVE TIMED OUT HERE !!!
[2023-09-26T11:07:49.362-0700] {dag.py:3683} INFO -
*****************************************************
[2023-09-26T11:07:49.362-0700] {dag.py:3687} INFO - Running task
external_dag_sensor
[2023-09-26 11:07:49,412] {taskinstance.py:1545} INFO - Exporting env vars:
AIRFLOW_CTX_DAG_OWNER='test' AIRFLOW_CTX_DAG_ID='DAG-1'
AIRFLOW_CTX_TASK_ID='external_dag_sensor'
AIRFLOW_CTX_EXECUTION_DATE='2023-09-26T18:01:48.838825+00:00'
AIRFLOW_CTX_TRY_NUMBER='-5'
AIRFLOW_CTX_DAG_RUN_ID='manual__2023-09-26T18:01:48.838825+00:00'
[2023-09-26T11:07:49.412-0700] {taskinstance.py:1545} INFO - Exporting env
vars: AIRFLOW_CTX_DAG_OWNER='test' AIRFLOW_CTX_DAG_ID='DAG-1'
AIRFLOW_CTX_TASK_ID='external_dag_sensor'
AIRFLOW_CTX_EXECUTION_DATE='2023-09-26T18:01:48.838825+00:00'
AIRFLOW_CTX_TRY_NUMBER='-5'
AIRFLOW_CTX_DAG_RUN_ID='manual__2023-09-26T18:01:48.838825+00:00'
[2023-09-26 11:07:49,417] {external_task.py:247} INFO - Poking for DAG
'some_dag' on 2023-09-26T18:01:48.838825+00:00 ...
[2023-09-26T11:07:49.417-0700] {external_task.py:247} INFO - Poking for DAG
'some_dag' on 2023-09-26T18:01:48.838825+00:00 ...
[2023-09-26 11:07:49,451] {taskinstance.py:1784} INFO - Rescheduling task,
marking task as UP_FOR_RESCHEDULE
[2023-09-26T11:07:49.451-0700] {taskinstance.py:1784} INFO - Rescheduling
task, marking task as UP_FOR_RESCHEDULE
[2023-09-26T11:07:49.452-0700] {dag.py:3691} INFO - external_dag_sensor ran
successfully!
[2023-09-26T11:07:49.452-0700] {dag.py:3694} INFO -
*****************************************************
[2023-09-26T11:08:49.428-0700] {dag.py:3683} INFO -
*****************************************************
[2023-09-26T11:08:49.429-0700] {dag.py:3687} INFO - Running task
external_dag_sensor
[2023-09-26 11:08:49,481] {taskinstance.py:1545} INFO - Exporting env vars:
AIRFLOW_CTX_DAG_OWNER='test' AIRFLOW_CTX_DAG_ID='DAG-1'
AIRFLOW_CTX_TASK_ID='external_dag_sensor'
AIRFLOW_CTX_EXECUTION_DATE='2023-09-26T18:01:48.838825+00:00'
AIRFLOW_CTX_TRY_NUMBER='-6'
AIRFLOW_CTX_DAG_RUN_ID='manual__2023-09-26T18:01:48.838825+00:00'
[2023-09-26T11:08:49.481-0700] {taskinstance.py:1545} INFO - Exporting env
vars: AIRFLOW_CTX_DAG_OWNER='test' AIRFLOW_CTX_DAG_ID='DAG-1'
AIRFLOW_CTX_TASK_ID='external_dag_sensor'
AIRFLOW_CTX_EXECUTION_DATE='2023-09-26T18:01:48.838825+00:00'
AIRFLOW_CTX_TRY_NUMBER='-6'
AIRFLOW_CTX_DAG_RUN_ID='manual__2023-09-26T18:01:48.838825+00:00'
[2023-09-26 11:08:49,487] {external_task.py:247} INFO - Poking for DAG
'some_dag' on 2023-09-26T18:01:48.838825+00:00 ...
[2023-09-26T11:08:49.487-0700] {external_task.py:247} INFO - Poking for DAG
'some_dag' on 2023-09-26T18:01:48.838825+00:00 ...
[2023-09-26 11:08:49,517] {taskinstance.py:1784} INFO - Rescheduling task,
marking task as UP_FOR_RESCHEDULE
[2023-09-26T11:08:49.517-0700] {taskinstance.py:1784} INFO - Rescheduling
task, marking task as UP_FOR_RESCHEDULE
[2023-09-26T11:08:49.518-0700] {dag.py:3691} INFO - external_dag_sensor ran
successfully!
[2023-09-26T11:08:49.518-0700] {dag.py:3694} INFO -
*****************************************************
[2023-09-26T11:09:49.504-0700] {dag.py:3683} INFO -
*****************************************************
[2023-09-26T11:09:49.505-0700] {dag.py:3687} INFO - Running task
external_dag_sensor
[2023-09-26 11:09:49,566] {taskinstance.py:1545} INFO - Exporting env vars:
AIRFLOW_CTX_DAG_OWNER='test' AIRFLOW_CTX_DAG_ID='DAG-1'
AIRFLOW_CTX_TASK_ID='external_dag_sensor'
AIRFLOW_CTX_EXECUTION_DATE='2023-09-26T18:01:48.838825+00:00'
AIRFLOW_CTX_TRY_NUMBER='-7'
AIRFLOW_CTX_DAG_RUN_ID='manual__2023-09-26T18:01:48.838825+00:00'
[2023-09-26T11:09:49.566-0700] {taskinstance.py:1545} INFO - Exporting env
vars: AIRFLOW_CTX_DAG_OWNER='test' AIRFLOW_CTX_DAG_ID='DAG-1'
AIRFLOW_CTX_TASK_ID='external_dag_sensor'
AIRFLOW_CTX_EXECUTION_DATE='2023-09-26T18:01:48.838825+00:00'
AIRFLOW_CTX_TRY_NUMBER='-7'
AIRFLOW_CTX_DAG_RUN_ID='manual__2023-09-26T18:01:48.838825+00:00'
[2023-09-26 11:09:49,572] {external_task.py:247} INFO - Poking for DAG
'some_dag' on 2023-09-26T18:01:48.838825+00:00 ...
[2023-09-26T11:09:49.572-0700] {external_task.py:247} INFO - Poking for DAG
'some_dag' on 2023-09-26T18:01:48.838825+00:00 ...
[2023-09-26 11:09:49,599] {taskinstance.py:1784} INFO - Rescheduling task,
marking task as UP_FOR_RESCHEDULE
[2023-09-26T11:09:49.599-0700] {taskinstance.py:1784} INFO - Rescheduling
task, marking task as UP_FOR_RESCHEDULE
[2023-09-26T11:09:49.600-0700] {dag.py:3691} INFO - external_dag_sensor ran
successfully!
[2023-09-26T11:09:49.600-0700] {dag.py:3694} INFO -
*****************************************************
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
