dstandish commented on code in PR #43520:
URL: https://github.com/apache/airflow/pull/43520#discussion_r1844674828
##########
airflow/jobs/scheduler_job_runner.py:
##########
@@ -1767,48 +1780,137 @@ def _send_dag_callbacks_to_processor(self, dag: DAG,
callback: DagCallbackReques
self.log.debug("callback is empty")
@provide_session
- def _fail_tasks_stuck_in_queued(self, session: Session = NEW_SESSION) ->
None:
+ def _handle_tasks_stuck_in_queued(self, session: Session = NEW_SESSION) ->
None:
"""
- Mark tasks stuck in queued for longer than `task_queued_timeout` as
failed.
+
+ Handle the scenario where a task is queued for longer than
`task_queued_timeout`.
Tasks can get stuck in queued for a wide variety of reasons (e.g.
celery loses
track of a task, a cluster can't further scale up its workers, etc.),
but tasks
- should not be stuck in queued for a long time. This will mark tasks
stuck in
- queued for longer than `self._task_queued_timeout` as failed. If the
task has
- available retries, it will be retried.
+ should not be stuck in queued for a long time.
+
+ Originally, we simply marked a task as failed when it was stuck in
queued for
+ too long. We found that this led to suboptimal outcomes as ideally we
would like "failed"
+ to mean that a task was unable to run, instead of it meaning that we
were unable to run the task.
+
+ As a compromise between always failing a stuck task and always
rescheduling a stuck task (which could
Review Comment:
yea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
