Please remember that with the LocalExecutor your tasks run in process(group) with the scheduler. If you want to restart the scheduler, it will need to wait until all tasks have finished that are currently running. In addition if you tasks are resource intensive (cpu, memory) this can also affect the scheduler. In 1.9.0 we are a little bit more robust in this respect, but guarding against OOM errors is very hard.
Furthermore, the new logging framework in 1.9.0, will allow you to have logs centrally which might be convenient. However, documentation is not up to date so you will have to tune it yourself. My 2 cents, Bolke. > On 2 Nov 2017, at 18:55, Shoumitra Srivastava <[email protected]> wrote: > > Hi guys, > > So far we have had a lot of success testing out Airflow and we are now > going for a full scale deployment. To that end, we are considering > dockerizing airflow and deploying it on one of our ECS clusters. We are > planning on separating out the web server and the scheduler to separate > tasks and using local executor with an RDS postgres and redis backend. Does > anyone else have any suggestions regarding the setup? Any design patterns > or good practises and gotchas would be welcome. > > -Shoumitra
