tirkarthi commented on issue #64620:
URL: https://github.com/apache/airflow/issues/64620#issuecomment-4181622378
We also started seeing similar failures in production. One of our
environments has 400-600 triggers running at a time. We started noticing this
in `ExternalTaskSensor` in deferred mode with default poll_interval of 2
seconds. The tasks used to run for days waiting for the external task to
finish. After Airflow 3 upgrade this runs for few hours like 3 to 7 hours and
fails randomly. Full traceback as below. For now we recommended increasing the
poll_interval to 300 seconds or set deferrable=False since these are long
running jobs to see if it helps.
We also have other environment in production where the same sensor runs but
the running triggers are around 10-20 at a time. We noticed this occurring once
or twice maybe there. So this could be due to high load and something our
non-production didn't catch due to lack of load in non-production.
```
[2026-04-02 21:33:12] ERROR - Trigger failed:
Traceback (most recent call last):
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py",
line 999, in cleanup_finished_triggers
result = details["task"].result()
^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/greenback/_impl.py",
line 116, in greenback_shim
return await _greenback_shim(orig_coro, next_send) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/greenback/_impl.py",
line 201, in _greenback_shim
next_yield, resume_greenlet = resume_greenlet.switch(next_send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/greenback/_impl.py",
line 81, in trampoline
next_yield: Any = next_send.send(orig_coro) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/outcome/_impl.py",
line 185, in send
return gen.send(self.value)
^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py",
line 1113, in run_trigger
async for event in trigger.run():
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/providers/standard/triggers/external_task.py",
line 125, in run
allowed_count = await get_count_func(self.allowed_states)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/providers/standard/triggers/external_task.py",
line 138, in _get_count_af_3
count = await sync_to_async(RuntimeTaskInstance.get_ti_count)(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/asgiref/sync.py",
line 504, in __call__
ret = await asyncio.shield(exec_coro)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/greenback/_impl.py",
line 210, in _greenback_shim
next_send = outcome.Value((yield next_yield))
^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/airflow_app/lib/python3.12/concurrent/futures/thread.py", line
58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/asgiref/sync.py",
line 559, in thread_handler
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py",
line 502, in get_ti_count
response = SUPERVISOR_COMMS.send(
^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py",
line 770, in send
return async_to_sync(self.asend)(msg)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/asgiref/sync.py",
line 325, in __call__
return call_result.result()
^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/airflow_app/lib/python3.12/concurrent/futures/_base.py", line
449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/airflow_app/lib/python3.12/concurrent/futures/_base.py", line
401, in __get_result
raise self._exception
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/asgiref/sync.py",
line 365, in main_wrap
result = await awaitable
^^^^^^^^^^^^^^^
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py",
line 799, in asend
return await self._aget_response(frame.id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py",
line 789, in _aget_response
raise RuntimeError(f"Response read out of order! Got {frame.id=},
{expect_id=}")
[2026-04-02 21:33:12] ERROR - Task failed with exception
TaskDeferralError: Trigger failure
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py",
line 953 in run
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py",
line 1344 in _execute_task
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/sdk/bases/sensor.py",
line 255 in resume_execution
File
"/opt/conda/envs/airflow_app/lib/python3.12/site-packages/airflow/sdk/bases/operator.py",
line 1630 in resume_execution
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
