My $.02 - posted to SO as well.
I fought this use case for a long time. In short, a dag that’s built based on
the state of a changing resource, especially a db table, doesn’t fly so well in
airflow.
My solution was to write a small custom operator that’s a subclass if
truggerdagoperator, it
To my mind, it's not a great idea to clear a resource that you're
dynamically using to determine the contents of a DAG. Is there any way that
you can refactor the table to be immutable? Instead of querying all rows in
the table, you would query records in an "unprocessed" state. Instead of
Question for the community. Did some hunting around and didn't see any
compelling answers. SO link:
https://stackoverflow.com/questions/49290546/how-to-set-up-a-dag-when-downstream-task-definitions-depend-on-upstream-outcomes
--
*Aaron Polhamus*
*Chief Technology Officer *
Cel (México): +52
