SamWheating commented on pull request #17963:
URL: https://github.com/apache/airflow/pull/17963#issuecomment-910531103
Looks great!
In the future, how would you feel about adding support for waiting for DAGs
to update. This could take shape as an operator which either:
- Forces the re-parsing of a file by spinning up a DagFileProcessor on the
worker (this is kinda hacky and probably won't work with all executors)
- Polls the `last_parsed_time` of a DAG and waits until it updates before
continuing (this also might not work if there's filesystem latency).
The idea being that we could build one DAG from the result of a task, wait
for it to re-parse and then trigger it, which would provide a more stable
solution for people trying to provision tasks from the result of a previous DAG
or task, which seems to be a common issue:
https://stackoverflow.com/questions/48589633/airflow-dynamic-tasks-at-runtime
https://stackoverflow.com/questions/52558018/airflow-generate-dynamic-tasks-in-single-dag-task-n1-is-dependent-on-taskn
This is probably a larger conversation, but I'd be happy to help with
improved support for runtime-generated DAGs. Let me know what you think 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
