o-nikolas commented on code in PR #26400:
URL: https://github.com/apache/airflow/pull/26400#discussion_r1002192571
##########
docs/apache-airflow/executor/debug.rst:
##########
@@ -15,11 +15,59 @@
specific language governing permissions and limitations
under the License.
+Testing DAGs with dag.test()
+=============================
+
+To debug DAGs in an IDE, you can set up the ``dag.test`` command in your dag
file and run through your DAG in a single
+serialized python process.
+
+This approach can be used with any supported database (including a local
SQLite database) and will
+*fail fast* as all tasks run in a single process.
+
+To set up ``dag.test``, add these two lines to the bottom of your dag file:
+
+.. code-block:: python
+
+ if __name__ == "__main__":
+ dag.test()
+
+and that's it! You can add argument such as ``execution_date`` if you want to
test argument-specific dagruns, but otherwise
+you can run or debug DAGs as needed.
+
+Comparison with DebugExecutor
+*****************************
+
+The ``dag.test`` command has the following benefits over the
:class:`~airflow.executors.debug_executor.DebugExecutor`
+class, which is now deprecated:
+
+1. It does not require running an executor at all. Tasks are run one at a time
with no executor or scheduler logs.
+2. It is significantly faster than running code with a DebugExecutor as it
does not need to go through a scheduler loop.
+3. It does not perform a backfill.
+
+
+Debugging Airflow DAGs on the command line
+==========================================
+
+With the same two line addition as mentioned in the above section, you can now
easily debug a DAG using pdb as well.
+Run ``python -m pdb <path to dag file>.py`` for an interactive debugging
experience on the command line.
+
+.. code-block:: bash
+
+ root@ef2c84ad4856:/opt/airflow# python -m pdb
airflow/example_dags/example_bash_operator.py
+ > /opt/airflow/airflow/example_dags/example_bash_operator.py(18)<module>()
+ -> """Example DAG demonstrating the usage of the BashOperator."""
+ (Pdb) b 45
+ Breakpoint 1 at /opt/airflow/airflow/example_dags/example_bash_operator.py:45
+ (Pdb) c
+ > /opt/airflow/airflow/example_dags/example_bash_operator.py(45)<module>()
+ -> bash_command='echo 1',
+ (Pdb) run_this_last
+ <Task(EmptyOperator): run_this_last>
.. _executor:DebugExecutor:
-Debug Executor
-==================
+Debug Executor (deprecated)
Review Comment:
+1 to @eladkal question.
I stumbled across this while working on something entirely different. I'm
very curious to have this discussion as well.
Currently
[AIP-47](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-47+New+design+of+Airflow+System+Tests)
system tests depend on the DebugExecutor:
> The tests should be structured in the way that they are easy to run as
“standalone” tests manually but they should also nicely be integrated into
pytest test execution environment. This can be achieved by leveraging the
**DebugExecutor** and utilising modern pytest test discovery mechanism
So we'd need to do some migration there if we were to deprecate this
executor (CC @potiuk @mnojek). I know @bhirsz has mentioned before they're
using a different executor altogether for their system tests, maybe that's a
more viable option.
We should also add deprecation warnings if we plan to deprecate this
executor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
