Hi, I think Jarek that you can unit test your DAGs without a BDD. You will
have to patching some connections but it's feasible. On my side I test the
topological order of my DAGs to be sure of the order.
And I patch the xcom_push and xcom_push methods to be sure that between
task everything will
Thanks! I like the suggestion about testing hooks rather than whole DAGs -
we will certainly use it in the future. And BDD is the approach I really
like - thanks for the code examples! We might also use it in the near
future. Super helpful!
So far we mocked hooks in our unit tests only (for
I have pylint set up in my IDE which catches most silly errors like missing
imports
I also use a docker image so I can start up airflow locally and manually
test any changes before trying to deploy them. I use a slightly modified
version of https://github.com/puckel/docker-airflow to control it.
There was a discussion about a unit testing approach last year 2017 I
believe. If you dig the mail archives, you can find it.
My take is:
- You should test "hooks" against some real system, which can be a docker
container. Make sure the behavior is predictable when talking against that
system.
I think it would be great to have a way to mock airflow for unit tests. The way
I approached this was to create a context manager that creates a temporary
directory, sets the AIRFLOW_HOME environment variable to this directory (only
within the scope of the context manager) and then renders an
