potiuk commented on code in PR #71457: URL: https://github.com/apache/airflow/pull/71457#discussion_r4175729045
########## airflow-core/docs/troubleshooting.rst: ########## @@ -20,6 +20,108 @@ Troubleshooting =============== +How to debug your Airflow deployment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The sections below walk through Airflow deployment issues using David A. Wheeler's writeup +of Agans' nine rules of debugging [1]_, with an Airflow-specific action for each rule. They are +written as a general starting point; for the specific known failure modes already documented, +see `Obscure task failures`_ below. + +Understand the system +---------------------- + +A minimal Airflow deployment is made up of a *scheduler*, a *Dag processor*, a *Dag bundle*, +an *API server*, and a *metadata database*; larger deployments add *workers* and a *triggerer*. +Each component can fail independently, and Airflow 3 removed the standalone webserver process +in favor of the API server, and moved Dag parsing out of the scheduler and into its own +*Dag processor* process. Before debugging a specific failure, know which of these components +is involved. See :doc:`/core-concepts/overview` for the full component breakdown. + +Make it fail +------------ + +Reproduce the failure outside of the full scheduling loop before you start changing things: + +- ``airflow tasks test <dag_id> <task_id> [logical_date_or_run_id]`` runs a single task instance + without checking dependencies or recording state in the database. Review Comment: **`airflow tasks test` does write to the DB in Airflow 3.** The CLI help text still says "without ... recording its state in the database", but `task_test` uses `_get_ti(..., create_if_necessary="db")`. With an existing run id it picks up the real task instance, sets it to `QUEUED` and runs it through the in-process Execution API, so the final state is persisted. Only a Dag run the command created itself is deleted afterwards. If someone re-runs `airflow tasks test <dag> <task> <real run_id>` against a production run, it overwrites that task instance's state. Could you describe it accurately and recommend omitting the run id, so a throw-away run is created and cleaned up? It's also worth noting that the command needs the Dag to be serialized already (it reads it from the DB), so it won't help when the Dag processor can't parse the file. --- Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting ########## airflow-core/docs/troubleshooting.rst: ########## @@ -20,6 +20,108 @@ Troubleshooting =============== +How to debug your Airflow deployment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The sections below walk through Airflow deployment issues using David A. Wheeler's writeup +of Agans' nine rules of debugging [1]_, with an Airflow-specific action for each rule. They are +written as a general starting point; for the specific known failure modes already documented, +see `Obscure task failures`_ below. + +Understand the system +---------------------- + +A minimal Airflow deployment is made up of a *scheduler*, a *Dag processor*, a *Dag bundle*, +an *API server*, and a *metadata database*; larger deployments add *workers* and a *triggerer*. +Each component can fail independently, and Airflow 3 removed the standalone webserver process +in favor of the API server, and moved Dag parsing out of the scheduler and into its own +*Dag processor* process. Before debugging a specific failure, know which of these components +is involved. See :doc:`/core-concepts/overview` for the full component breakdown. + +Make it fail +------------ + +Reproduce the failure outside of the full scheduling loop before you start changing things: + +- ``airflow tasks test <dag_id> <task_id> [logical_date_or_run_id]`` runs a single task instance + without checking dependencies or recording state in the database. +- ``airflow dags test <dag_id> [logical_date]`` runs one full DagRun locally, without the + scheduler. + +Both commands run in your current process, so you can also attach a debugger to them; see +:doc:`/core-concepts/debug` for running a Dag under ``pdb`` or an IDE debugger. + +Quit thinking and look +----------------------- + +Read the actual task log before guessing at a cause. By default, task logs are written under +``$AIRFLOW_HOME/logs/`` using the path Review Comment: The log root is `[logging] base_log_folder` (defaulting to `$AIRFLOW_HOME/logs`), on the worker that ran the task, and with remote logging possibly only in the remote store. Worth mentioning alongside `log_filename_template`. --- Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting ########## airflow-core/docs/troubleshooting.rst: ########## @@ -20,6 +20,108 @@ Troubleshooting =============== +How to debug your Airflow deployment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The sections below walk through Airflow deployment issues using David A. Wheeler's writeup +of Agans' nine rules of debugging [1]_, with an Airflow-specific action for each rule. They are +written as a general starting point; for the specific known failure modes already documented, +see `Obscure task failures`_ below. + +Understand the system +---------------------- + +A minimal Airflow deployment is made up of a *scheduler*, a *Dag processor*, a *Dag bundle*, +an *API server*, and a *metadata database*; larger deployments add *workers* and a *triggerer*. +Each component can fail independently, and Airflow 3 removed the standalone webserver process +in favor of the API server, and moved Dag parsing out of the scheduler and into its own +*Dag processor* process. Before debugging a specific failure, know which of these components +is involved. See :doc:`/core-concepts/overview` for the full component breakdown. + +Make it fail +------------ + +Reproduce the failure outside of the full scheduling loop before you start changing things: + +- ``airflow tasks test <dag_id> <task_id> [logical_date_or_run_id]`` runs a single task instance + without checking dependencies or recording state in the database. +- ``airflow dags test <dag_id> [logical_date]`` runs one full DagRun locally, without the + scheduler. + +Both commands run in your current process, so you can also attach a debugger to them; see +:doc:`/core-concepts/debug` for running a Dag under ``pdb`` or an IDE debugger. + +Quit thinking and look +----------------------- + +Read the actual task log before guessing at a cause. By default, task logs are written under +``$AIRFLOW_HOME/logs/`` using the path +``dag_id=<dag_id>/run_id=<run_id>/task_id=<task_id>/attempt=<n>.log`` (add a +``map_index=<n>/`` segment for mapped tasks). This is controlled by the +:ref:`logging.log_filename_template <config:logging__log_filename_template>` setting, so check +that setting if logs aren't where you expect them. ``airflow tasks state <dag_id> <task_id> +<logical_date_or_run_id>`` will confirm the recorded state of a task instance before you go +looking at logs at all. + +Divide and conquer +------------------- + +Narrow the failure down to a single component before digging further: + +- ``airflow dags list-import-errors`` shows Dags the *Dag processor* failed to parse. A Dag + that fails to parse is a Dag-processor problem, not a scheduler problem, in Airflow 3. +- ``airflow db check`` confirms the metadata database is reachable. +- ``airflow tasks failed-deps <dag_id> <task_id> <logical_date_or_run_id>`` shows the unmet + dependencies that are keeping the scheduler from queuing a task instance. + +Change one thing at a time +--------------------------- + +When testing a fix, change a single variable and re-run ``airflow tasks test`` (or +``airflow dags test``) before layering on the next change. ``airflow config get-value <section> +<option>`` prints the effective value of a single configuration option, so you can confirm +exactly what changed between runs instead of assuming. + +Keep an audit trail +-------------------- + +Record what you tried and what happened. ``airflow version`` records the exact version you were +running; ``airflow dags show <dag_id> --save graph.png`` saves the task dependency graph for a Review Comment: Nit: `dags show --save` needs Graphviz installed — worth a short note. --- Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting ########## airflow-core/docs/troubleshooting.rst: ########## @@ -20,6 +20,108 @@ Troubleshooting =============== +How to debug your Airflow deployment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The sections below walk through Airflow deployment issues using David A. Wheeler's writeup +of Agans' nine rules of debugging [1]_, with an Airflow-specific action for each rule. They are +written as a general starting point; for the specific known failure modes already documented, +see `Obscure task failures`_ below. + +Understand the system +---------------------- + +A minimal Airflow deployment is made up of a *scheduler*, a *Dag processor*, a *Dag bundle*, +an *API server*, and a *metadata database*; larger deployments add *workers* and a *triggerer*. +Each component can fail independently, and Airflow 3 removed the standalone webserver process +in favor of the API server, and moved Dag parsing out of the scheduler and into its own +*Dag processor* process. Before debugging a specific failure, know which of these components +is involved. See :doc:`/core-concepts/overview` for the full component breakdown. + +Make it fail +------------ + +Reproduce the failure outside of the full scheduling loop before you start changing things: + +- ``airflow tasks test <dag_id> <task_id> [logical_date_or_run_id]`` runs a single task instance + without checking dependencies or recording state in the database. +- ``airflow dags test <dag_id> [logical_date]`` runs one full DagRun locally, without the + scheduler. + +Both commands run in your current process, so you can also attach a debugger to them; see +:doc:`/core-concepts/debug` for running a Dag under ``pdb`` or an IDE debugger. + +Quit thinking and look +----------------------- + +Read the actual task log before guessing at a cause. By default, task logs are written under +``$AIRFLOW_HOME/logs/`` using the path +``dag_id=<dag_id>/run_id=<run_id>/task_id=<task_id>/attempt=<n>.log`` (add a +``map_index=<n>/`` segment for mapped tasks). This is controlled by the +:ref:`logging.log_filename_template <config:logging__log_filename_template>` setting, so check +that setting if logs aren't where you expect them. ``airflow tasks state <dag_id> <task_id> +<logical_date_or_run_id>`` will confirm the recorded state of a task instance before you go +looking at logs at all. + +Divide and conquer +------------------- + +Narrow the failure down to a single component before digging further: + +- ``airflow dags list-import-errors`` shows Dags the *Dag processor* failed to parse. A Dag + that fails to parse is a Dag-processor problem, not a scheduler problem, in Airflow 3. +- ``airflow db check`` confirms the metadata database is reachable. Review Comment: **Workers don't use the metadata DB in Airflow 3.** `airflow db check` only applies to the scheduler, Dag processor and API server. Workers talk to the API server via `[core] execution_api_server_url`, so for them that's the connectivity to check. For overall component health, linking the health-check docs (`/api/v2/monitor/health`, `airflow jobs check`) would fit "Divide and conquer" nicely. Related: all the CLI commands in this section need metadata DB access. One sentence saying to run them from a host that has it (scheduler / API server), not from a worker, would avoid a confusing first failure. --- Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting ########## airflow-core/docs/troubleshooting.rst: ########## @@ -20,6 +20,108 @@ Troubleshooting =============== +How to debug your Airflow deployment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The sections below walk through Airflow deployment issues using David A. Wheeler's writeup +of Agans' nine rules of debugging [1]_, with an Airflow-specific action for each rule. They are +written as a general starting point; for the specific known failure modes already documented, +see `Obscure task failures`_ below. + +Understand the system +---------------------- + +A minimal Airflow deployment is made up of a *scheduler*, a *Dag processor*, a *Dag bundle*, +an *API server*, and a *metadata database*; larger deployments add *workers* and a *triggerer*. +Each component can fail independently, and Airflow 3 removed the standalone webserver process +in favor of the API server, and moved Dag parsing out of the scheduler and into its own +*Dag processor* process. Before debugging a specific failure, know which of these components +is involved. See :doc:`/core-concepts/overview` for the full component breakdown. + +Make it fail +------------ + +Reproduce the failure outside of the full scheduling loop before you start changing things: + +- ``airflow tasks test <dag_id> <task_id> [logical_date_or_run_id]`` runs a single task instance + without checking dependencies or recording state in the database. +- ``airflow dags test <dag_id> [logical_date]`` runs one full DagRun locally, without the + scheduler. + +Both commands run in your current process, so you can also attach a debugger to them; see +:doc:`/core-concepts/debug` for running a Dag under ``pdb`` or an IDE debugger. + +Quit thinking and look +----------------------- + +Read the actual task log before guessing at a cause. By default, task logs are written under +``$AIRFLOW_HOME/logs/`` using the path +``dag_id=<dag_id>/run_id=<run_id>/task_id=<task_id>/attempt=<n>.log`` (add a +``map_index=<n>/`` segment for mapped tasks). This is controlled by the +:ref:`logging.log_filename_template <config:logging__log_filename_template>` setting, so check +that setting if logs aren't where you expect them. ``airflow tasks state <dag_id> <task_id> +<logical_date_or_run_id>`` will confirm the recorded state of a task instance before you go +looking at logs at all. + +Divide and conquer +------------------- + +Narrow the failure down to a single component before digging further: + +- ``airflow dags list-import-errors`` shows Dags the *Dag processor* failed to parse. A Dag + that fails to parse is a Dag-processor problem, not a scheduler problem, in Airflow 3. +- ``airflow db check`` confirms the metadata database is reachable. +- ``airflow tasks failed-deps <dag_id> <task_id> <logical_date_or_run_id>`` shows the unmet + dependencies that are keeping the scheduler from queuing a task instance. + +Change one thing at a time +--------------------------- + +When testing a fix, change a single variable and re-run ``airflow tasks test`` (or +``airflow dags test``) before layering on the next change. ``airflow config get-value <section> +<option>`` prints the effective value of a single configuration option, so you can confirm +exactly what changed between runs instead of assuming. + +Keep an audit trail +-------------------- + +Record what you tried and what happened. ``airflow version`` records the exact version you were +running; ``airflow dags show <dag_id> --save graph.png`` saves the task dependency graph for a +Dag; ``airflow tasks states-for-dag-run <dag_id> <logical_date_or_run_id>`` records the state of +every task instance in a run. Keeping these alongside your notes makes it possible to tell later +whether a change actually affected behavior. + +Check the plug +--------------- + +Before debugging further, check the things that are easy to overlook: + +- Is the Dag paused? ``airflow dags list`` includes an ``is_paused`` column. +- Can the component you're debugging actually reach the metadata database? ``airflow db check``. Review Comment: Same as line 73: for a worker, the thing to check is reaching the API server (`[core] execution_api_server_url`), not the metadata database. --- Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting ########## airflow-core/docs/troubleshooting.rst: ########## @@ -20,6 +20,108 @@ Troubleshooting =============== +How to debug your Airflow deployment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The sections below walk through Airflow deployment issues using David A. Wheeler's writeup +of Agans' nine rules of debugging [1]_, with an Airflow-specific action for each rule. They are +written as a general starting point; for the specific known failure modes already documented, +see `Obscure task failures`_ below. + +Understand the system +---------------------- + +A minimal Airflow deployment is made up of a *scheduler*, a *Dag processor*, a *Dag bundle*, +an *API server*, and a *metadata database*; larger deployments add *workers* and a *triggerer*. +Each component can fail independently, and Airflow 3 removed the standalone webserver process +in favor of the API server, and moved Dag parsing out of the scheduler and into its own +*Dag processor* process. Before debugging a specific failure, know which of these components +is involved. See :doc:`/core-concepts/overview` for the full component breakdown. + +Make it fail +------------ + +Reproduce the failure outside of the full scheduling loop before you start changing things: + +- ``airflow tasks test <dag_id> <task_id> [logical_date_or_run_id]`` runs a single task instance + without checking dependencies or recording state in the database. +- ``airflow dags test <dag_id> [logical_date]`` runs one full DagRun locally, without the + scheduler. + +Both commands run in your current process, so you can also attach a debugger to them; see +:doc:`/core-concepts/debug` for running a Dag under ``pdb`` or an IDE debugger. + +Quit thinking and look +----------------------- + +Read the actual task log before guessing at a cause. By default, task logs are written under +``$AIRFLOW_HOME/logs/`` using the path +``dag_id=<dag_id>/run_id=<run_id>/task_id=<task_id>/attempt=<n>.log`` (add a +``map_index=<n>/`` segment for mapped tasks). This is controlled by the +:ref:`logging.log_filename_template <config:logging__log_filename_template>` setting, so check +that setting if logs aren't where you expect them. ``airflow tasks state <dag_id> <task_id> +<logical_date_or_run_id>`` will confirm the recorded state of a task instance before you go +looking at logs at all. + +Divide and conquer +------------------- + +Narrow the failure down to a single component before digging further: + +- ``airflow dags list-import-errors`` shows Dags the *Dag processor* failed to parse. A Dag + that fails to parse is a Dag-processor problem, not a scheduler problem, in Airflow 3. +- ``airflow db check`` confirms the metadata database is reachable. +- ``airflow tasks failed-deps <dag_id> <task_id> <logical_date_or_run_id>`` shows the unmet + dependencies that are keeping the scheduler from queuing a task instance. + +Change one thing at a time +--------------------------- + +When testing a fix, change a single variable and re-run ``airflow tasks test`` (or Review Comment: Given the comment on line 47, re-running `tasks test` with a real run id changes that run's task instance state — worth pointing people at the run-id-less form here (and at line 117). --- Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting ########## airflow-core/docs/troubleshooting.rst: ########## @@ -20,6 +20,108 @@ Troubleshooting =============== +How to debug your Airflow deployment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The sections below walk through Airflow deployment issues using David A. Wheeler's writeup +of Agans' nine rules of debugging [1]_, with an Airflow-specific action for each rule. They are +written as a general starting point; for the specific known failure modes already documented, +see `Obscure task failures`_ below. + +Understand the system +---------------------- + +A minimal Airflow deployment is made up of a *scheduler*, a *Dag processor*, a *Dag bundle*, +an *API server*, and a *metadata database*; larger deployments add *workers* and a *triggerer*. +Each component can fail independently, and Airflow 3 removed the standalone webserver process +in favor of the API server, and moved Dag parsing out of the scheduler and into its own +*Dag processor* process. Before debugging a specific failure, know which of these components +is involved. See :doc:`/core-concepts/overview` for the full component breakdown. + +Make it fail +------------ + +Reproduce the failure outside of the full scheduling loop before you start changing things: + +- ``airflow tasks test <dag_id> <task_id> [logical_date_or_run_id]`` runs a single task instance + without checking dependencies or recording state in the database. +- ``airflow dags test <dag_id> [logical_date]`` runs one full DagRun locally, without the Review Comment: Nit: "DagRun" in prose should be "Dag run". --- Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting ########## airflow-core/docs/troubleshooting.rst: ########## @@ -20,6 +20,108 @@ Troubleshooting =============== +How to debug your Airflow deployment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The sections below walk through Airflow deployment issues using David A. Wheeler's writeup +of Agans' nine rules of debugging [1]_, with an Airflow-specific action for each rule. They are +written as a general starting point; for the specific known failure modes already documented, +see `Obscure task failures`_ below. + +Understand the system +---------------------- + +A minimal Airflow deployment is made up of a *scheduler*, a *Dag processor*, a *Dag bundle*, +an *API server*, and a *metadata database*; larger deployments add *workers* and a *triggerer*. +Each component can fail independently, and Airflow 3 removed the standalone webserver process +in favor of the API server, and moved Dag parsing out of the scheduler and into its own +*Dag processor* process. Before debugging a specific failure, know which of these components +is involved. See :doc:`/core-concepts/overview` for the full component breakdown. + +Make it fail +------------ + +Reproduce the failure outside of the full scheduling loop before you start changing things: + +- ``airflow tasks test <dag_id> <task_id> [logical_date_or_run_id]`` runs a single task instance + without checking dependencies or recording state in the database. +- ``airflow dags test <dag_id> [logical_date]`` runs one full DagRun locally, without the + scheduler. + +Both commands run in your current process, so you can also attach a debugger to them; see +:doc:`/core-concepts/debug` for running a Dag under ``pdb`` or an IDE debugger. + +Quit thinking and look +----------------------- + +Read the actual task log before guessing at a cause. By default, task logs are written under +``$AIRFLOW_HOME/logs/`` using the path +``dag_id=<dag_id>/run_id=<run_id>/task_id=<task_id>/attempt=<n>.log`` (add a +``map_index=<n>/`` segment for mapped tasks). This is controlled by the +:ref:`logging.log_filename_template <config:logging__log_filename_template>` setting, so check +that setting if logs aren't where you expect them. ``airflow tasks state <dag_id> <task_id> +<logical_date_or_run_id>`` will confirm the recorded state of a task instance before you go +looking at logs at all. + +Divide and conquer +------------------- + +Narrow the failure down to a single component before digging further: + +- ``airflow dags list-import-errors`` shows Dags the *Dag processor* failed to parse. A Dag + that fails to parse is a Dag-processor problem, not a scheduler problem, in Airflow 3. +- ``airflow db check`` confirms the metadata database is reachable. +- ``airflow tasks failed-deps <dag_id> <task_id> <logical_date_or_run_id>`` shows the unmet + dependencies that are keeping the scheduler from queuing a task instance. + +Change one thing at a time +--------------------------- + +When testing a fix, change a single variable and re-run ``airflow tasks test`` (or +``airflow dags test``) before layering on the next change. ``airflow config get-value <section> +<option>`` prints the effective value of a single configuration option, so you can confirm +exactly what changed between runs instead of assuming. + +Keep an audit trail +-------------------- + +Record what you tried and what happened. ``airflow version`` records the exact version you were +running; ``airflow dags show <dag_id> --save graph.png`` saves the task dependency graph for a +Dag; ``airflow tasks states-for-dag-run <dag_id> <logical_date_or_run_id>`` records the state of +every task instance in a run. Keeping these alongside your notes makes it possible to tell later +whether a change actually affected behavior. + +Check the plug +--------------- + +Before debugging further, check the things that are easy to overlook: + +- Is the Dag paused? ``airflow dags list`` includes an ``is_paused`` column. +- Can the component you're debugging actually reach the metadata database? ``airflow db check``. +- Are you confusing the Dag's *logical date* with wall-clock time? A DagRun's ``logical_date`` Review Comment: Nit: "DagRun" in prose should be "Dag run". --- Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
