What is a good example of a testing flow for new/updated DAGs? The "airflow test" CLI command is great for testing individual tasks, but for testing task interactions and the actual DAG topology (especially things like branching or skipping) one needs to use the airflow scheduler. It seems to me there are a few options here:
1. Start up an airflow scheduler locally, running with sqlite and SequentialExecutor. Probably not production-like, but trivial to set up. 2. Deploy a more production-like airflow scheduler with the DAG you're testing. Gives a better idea of how your code'll behave in production, but feels like a pretty heavyweight testing solution. 3. Somehow submit your DAG-under-test to the production scheduler, watch it run, edit and resubmit it if necessary, and remove it -- all without disrupting the scheduler's other in-flight work. Gives the best preview of your code in production, and low operational overhead for the dev. #2 is mainly what I've been doing. #3 seems best, but I'm not immediately aware of a clean way to add/update/remove DAGs in a remotely hosted scheduler in an ad-hoc fashion? Hopefully most of this makes sense, but please point out gaps in my understanding!
