Note - for DebugExecutor you do not need webserver nor UI (in fact it does not even use the DB to record the runs so you wouldn't see anything there).
On Thu, Apr 14, 2022 at 8:25 PM Jarek Potiuk <[email protected]> wrote: > > Hey Alexander, > > Airflow does not work like that for "regular" use - but you can use > DebugExecutor which does what you want > https://airflow.apache.org/docs/apache-airflow/stable/executor/debug.html?highlight=debugexecutor > > But note it's just for debugging. Airflow is not supposed to be used > to trigger dags immediately - it's a scheduler and it's optimized for > that. We do have discussions about enabling such cases in the future > but this not were Airflow use case is today. It's used for scheduling. > > J > > On Thu, Apr 14, 2022 at 8:02 PM Daniel Standish > <[email protected]> wrote: > > > > I see. Ok well yeah, the scheduler won't register the dag immediately. > > There are settings which control how frequently it checks for new files. > > > > You might try shortening the time for this. I don't know the setting off > > top of head but look at the config. > > > > You could consider using the API to check whether the dag was picked up > > before triggering: > > https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_dag > > > > You could also consider not writing out the dag file dynamically in this > > way and instead use "dag factory" functions, or task groups -- these are > > sometimes used for similar goals. And in 2.3.0 (beta coming soon) there > > will be support for task mapping which can also achieve similar things.
