SamWheating commented on pull request #17963:
URL: https://github.com/apache/airflow/pull/17963#issuecomment-910417229
I've encountered some race-conditions with this kind of dynamic task
generation before, typically when trying to update the metadata file and
trigger the DAG immediately after.
Two cases where this shows up:
- DAG_1 writes a metadata file and triggers DAG 2 (which uses the metadata
file to generate tasks)
- Uploading a metadata file and immediately triggering a DAG via REST API
In these cases, the DAG often isn't re-processed between the time that the
metadata file is updated and the time that the DAG is triggered, so the tasks
which are created are based on the previous version of the metadata file.
I think that this can be a bit of a trap and can definitely be misleading
for users who aren't familiar with Airflow's internals. Do you think we can
include some warnings about this behaviour in the documentation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
