We have been using airflow for logic that delegates to other systems so
inject a task all tasks depends to make sure all resources used are the
same for all tasks in the dag. This works well for tasks that delegates to
external systems but people are starting to need to run logic in airflow
and the fact that scheduler and all workers can see different states is
causing issues

We can make sure that all the code is deployed in a consistent way but need
help from the scheduler to tell the workers the current generation for a
DAG.

My question is, what would be the best way to modify airflow to allow DAGs
to define a generation value that the scheduler could send to workers?

Thanks

Reply via email to