I have pylint set up in my IDE which catches most silly errors like missing
imports
I also use a docker image so I can start up airflow locally and manually
test any changes before trying to deploy them. I use a slightly modified
version of https://github.com/puckel/docker-airflow to control it.
Hi,
I am Ahlem Triki, software Engineer. Recently we started working with
Airflow and I would ask if you see Airflow as a fit to solve what we working
on, or is it an over kill. We are really in need for an airflow expert point
of view.
Here is a brief description of what we working on:
*
We use Airflow for a mixture of traditional data pipeline work, and
orchestration tasks that need to express complex date and dependency logic.
The former is a bit better supported, but it's still a great tool for the
latter.
But it sounds like your format is "this task should happen whenever
There was a discussion about a unit testing approach last year 2017 I
believe. If you dig the mail archives, you can find it.
My take is:
- You should test "hooks" against some real system, which can be a docker
container. Make sure the behavior is predictable when talking against that
system.
