Re: slow scheduler

2018-04-04 Thread Maxime Beauchemin
As a batch scheduler Airflow doesn't currently guarantee super low latency. The aim for the project has been to make it possible to do sub-minute latency at scale, but it's common for this go up to a few minutes in larger environments. I'd recommend making it clear to your users what your Airflow

slow scheduler

2018-04-04 Thread Cieplucha, Michal
Hello all, Our automation with airflow is getting bigger and bigger (airflow 1.8, ~150 DAGs, 3xinstances of scheduler) . Sometimes our users are triggering DAG runs based on some external events, so we exposed an API endpoint to run a DAG. Those DAGs that are run manually should give fast feedb