ucaeon commented on code in PR #72492: URL: https://github.com/apache/airflow/pull/72492#discussion_r3978497416
########## airflow-core/docs/troubleshooting.rst: ########## @@ -26,15 +26,84 @@ Obscure task failures Task state changed externally ----------------------------- -There are many potential causes for a task's state to be changed by a component other than the executor, which might cause some confusion when reviewing task instance or scheduler logs. +This message indicates that the task instance's state does not match the state reported by another component. The message itself does not identify the root cause. -Below are some example scenarios that could cause a task's state to change by a component other than the executor: +What this message means +""""""""""""""""""""""" + +Task state can be updated by different Airflow components or by +external actions. If the state reported by the executor does not match +the state stored for the task instance, Airflow may log a state mismatch. + +One common message looks like this:: + + Executor <executor> reported that the task instance <ti> finished with state <executor_state>, but the task instance's state attribute is <ti_state>. + +Check the logs around the time of the state change to determine what caused it. + +How to investigate +"""""""""""""""""" + +Start with the task and scheduler logs, then check the worker or +infrastructure logs if the cause is not clear. + +1. Check task logs. If there are no logs in the UI, the task likely never started on a worker, or the worker died before it could write logs. +2. Check scheduler logs around the same timestamp. +3. Check worker or executor logs, to see whether the task was received, started, or never dispatched. +4. Check infrastructure logs (container, pod, or host) for OOM, eviction, or restarts. See :ref:`troubleshooting-process-terminated-by-signal` for SIGTERM and SIGKILL. +5. Check whether a user or an external process changed the task state in the Airflow UI or through the :doc:`Airflow REST API <stable-rest-api-ref>`. + +Common causes +""""""""""""" + +Below are some example scenarios where a task's state may be changed by a component other than the executor: - If a task's Dag failed to parse on the worker, the scheduler may mark the task as failed. If confirmed, consider increasing :ref:`core.dagbag_import_timeout <config:core__dagbag_import_timeout>` and :ref:`dag_processor.dag_file_processor_timeout <config:dag_processor__dag_file_processor_timeout>`. -- The scheduler will mark a task as failed if the task has been queued for longer than :ref:`scheduler.task_queued_timeout <config:scheduler__task_queued_timeout>`. -- If a :ref:`task instance's heartbeat times out <concepts:task-instance-heartbeat-timeout>`, it will be marked failed by the scheduler. -- A user marked the task as successful or failed in the Airflow UI. -- An external script or process used the :doc:`Airflow REST API <stable-rest-api-ref>` to change the state of a task. +- A task can be retried or marked as failed if it remains queued longer than :ref:`scheduler.task_queued_timeout <config:scheduler__task_queued_timeout>`. See :ref:`troubleshooting-task-stuck-queued`. +- If a :ref:`task instance's heartbeat times out <concepts:task-instance-heartbeat-timeout>`, it will be marked failed by the scheduler. See :ref:`troubleshooting-task-stuck-running`. +- The task process was killed by the operating system or orchestrator. See :ref:`troubleshooting-process-terminated-by-signal`. +- A user or an external process marked the task as successful or failed in the Airflow UI or through the :doc:`Airflow REST API <stable-rest-api-ref>`. + +.. _troubleshooting-task-stuck-queued: + +Task stuck in queued state +-------------------------- + +A task remains ``queued`` while it is waiting to be executed. If it stays queued longer than :ref:`scheduler.task_queued_timeout <config:scheduler__task_queued_timeout>` (default 600 seconds), it may be retried or marked as failed. There will often be no task logs in the UI, because a worker never ran the task. + +Here are some of the common causes: + +- Not enough worker capacity, pool slots, or :ref:`core.parallelism <config:core__parallelism>`. +- The executor is unable to dispatch tasks to workers. +- The queued timeout is shorter than how long tasks wait under normal load. + +How to troubleshoot: + +- Check scheduler logs for tasks stuck in ``queued``. +- Confirm workers are running and accepting work. +- Check pool and concurrency limits. Review Comment: You're right—pool and concurrency limits are checked before a task enters the queued state. I've removed those references from both the common causes and troubleshooting steps. Thanks for catching this!😊 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
