baha-bouali commented on code in PR #71457: URL: https://github.com/apache/airflow/pull/71457#discussion_r4200235269
########## airflow-core/docs/troubleshooting.rst: ########## @@ -20,6 +20,108 @@ Troubleshooting =============== +How to debug your Airflow deployment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The sections below walk through Airflow deployment issues using David A. Wheeler's writeup +of Agans' nine rules of debugging [1]_, with an Airflow-specific action for each rule. They are +written as a general starting point; for the specific known failure modes already documented, +see `Obscure task failures`_ below. + +Understand the system +---------------------- + +A minimal Airflow deployment is made up of a *scheduler*, a *Dag processor*, a *Dag bundle*, +an *API server*, and a *metadata database*; larger deployments add *workers* and a *triggerer*. +Each component can fail independently, and Airflow 3 removed the standalone webserver process +in favor of the API server, and moved Dag parsing out of the scheduler and into its own +*Dag processor* process. Before debugging a specific failure, know which of these components +is involved. See :doc:`/core-concepts/overview` for the full component breakdown. + +Make it fail +------------ + +Reproduce the failure outside of the full scheduling loop before you start changing things: + +- ``airflow tasks test <dag_id> <task_id> [logical_date_or_run_id]`` runs a single task instance + without checking dependencies or recording state in the database. +- ``airflow dags test <dag_id> [logical_date]`` runs one full DagRun locally, without the + scheduler. + +Both commands run in your current process, so you can also attach a debugger to them; see +:doc:`/core-concepts/debug` for running a Dag under ``pdb`` or an IDE debugger. + +Quit thinking and look +----------------------- + +Read the actual task log before guessing at a cause. By default, task logs are written under +``$AIRFLOW_HOME/logs/`` using the path +``dag_id=<dag_id>/run_id=<run_id>/task_id=<task_id>/attempt=<n>.log`` (add a +``map_index=<n>/`` segment for mapped tasks). This is controlled by the +:ref:`logging.log_filename_template <config:logging__log_filename_template>` setting, so check +that setting if logs aren't where you expect them. ``airflow tasks state <dag_id> <task_id> +<logical_date_or_run_id>`` will confirm the recorded state of a task instance before you go +looking at logs at all. + +Divide and conquer +------------------- + +Narrow the failure down to a single component before digging further: + +- ``airflow dags list-import-errors`` shows Dags the *Dag processor* failed to parse. A Dag + that fails to parse is a Dag-processor problem, not a scheduler problem, in Airflow 3. +- ``airflow db check`` confirms the metadata database is reachable. +- ``airflow tasks failed-deps <dag_id> <task_id> <logical_date_or_run_id>`` shows the unmet + dependencies that are keeping the scheduler from queuing a task instance. + +Change one thing at a time +--------------------------- + +When testing a fix, change a single variable and re-run ``airflow tasks test`` (or +``airflow dags test``) before layering on the next change. ``airflow config get-value <section> +<option>`` prints the effective value of a single configuration option, so you can confirm +exactly what changed between runs instead of assuming. + +Keep an audit trail +-------------------- + +Record what you tried and what happened. ``airflow version`` records the exact version you were +running; ``airflow dags show <dag_id> --save graph.png`` saves the task dependency graph for a +Dag; ``airflow tasks states-for-dag-run <dag_id> <logical_date_or_run_id>`` records the state of +every task instance in a run. Keeping these alongside your notes makes it possible to tell later +whether a change actually affected behavior. + +Check the plug +--------------- + +Before debugging further, check the things that are easy to overlook: + +- Is the Dag paused? ``airflow dags list`` includes an ``is_paused`` column. +- Can the component you're debugging actually reach the metadata database? ``airflow db check``. +- Are you confusing the Dag's *logical date* with wall-clock time? A DagRun's ``logical_date`` Review Comment: Done! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
