Hello Community, Recently we had several people who complained (on slack) that airflow 2.1 is slow in scheduling tasks. After some discussion it usually turned out that those people were using SQLite + Sequential executor. I think it gives very bad impression to users. We even had one user who almost gave up Airflow seeing how slow it is in scheduling tasks (!).
I think while in Airflow 1.10 the difference was not as noticeable, Airflow 2 with Postgres/MySQL is lightning fast comparing to sqlite. It's like a different world. First time users might get a very bad impression when their first contact with Airflow is via sqlite + Sequential executor. Many people choose sqlite as their first choice when they try Airflow (Sqlite is generally seen as solid choice in many cases and people are afraid that setting up MySQL/Postgres might take them a lot of time to setup). However with current Docker-Compose quilck-start by Kamil it is already rather quick to set-up a working setup with Postgres. My idea is - why don't we make SQLITE "development-only" choice. That would require editable, development version of airflow to run and fail hard when it is installed as regular package (with appropriate "Use proper database - MySQL/Postgres" - and MSSQL when we release MSSQL-support in 2.2 ). I think that would be possible, it would not violate backwards compatibility (sqlite was anyhow for development-only) and it would help Airflow with being seen as more "snappy". Any other ideas? WDYT? J. -- +48 660 796 129
