I see. Ok well yeah, the scheduler won't register the dag immediately. There are settings which control how frequently it checks for new files.
You might try shortening the time for this. I don't know the setting off top of head but look at the config. You could consider using the API to check whether the dag was picked up before triggering: https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_dag You could also consider not writing out the dag file dynamically in this way and instead use "dag factory" functions, or task groups -- these are sometimes used for similar goals. And in 2.3.0 (beta coming soon) there will be support for task mapping which can also achieve similar things.
