Some thought as a user of Airflow, I wouldn’t of adopted Airflow in the first place if I couldn’t test it with sqlite. And would be the same today, accessing docker isn’t always an easy in some companies.
But having a warning that sqlite is development only and much slower than other solutions when it’s enabled seems fair. Also forcing new time users to edit the config on first run I think is acceptable as they will need to get used to do that frequently anyway if they’re rolling their own install. Damian From: Jarek Potiuk <[email protected]> Sent: Tuesday, July 20, 2021 12:23 To: [email protected] Subject: [DISCUSSION] Should we be more explicit about SQLite using for dev only (or kill it for non-dev entirely????) Hello Community, Recently we had several people who complained (on slack) that airflow 2.1 is slow in scheduling tasks. After some discussion it usually turned out that those people were using SQLite + Sequential executor. I think it gives very bad impression to users. We even had one user who almost gave up Airflow seeing how slow it is in scheduling tasks (!). I think while in Airflow 1.10 the difference was not as noticeable, Airflow 2 with Postgres/MySQL is lightning fast comparing to sqlite. It's like a different world. First time users might get a very bad impression when their first contact with Airflow is via sqlite + Sequential executor. Many people choose sqlite as their first choice when they try Airflow (Sqlite is generally seen as solid choice in many cases and people are afraid that setting up MySQL/Postgres might take them a lot of time to setup). However with current Docker-Compose quilck-start by Kamil it is already rather quick to set-up a working setup with Postgres. My idea is - why don't we make SQLITE "development-only" choice. That would require editable, development version of airflow to run and fail hard when it is installed as regular package (with appropriate "Use proper database - MySQL/Postgres" - and MSSQL when we release MSSQL-support in 2.2 ). I think that would be possible, it would not violate backwards compatibility (sqlite was anyhow for development-only) and it would help Airflow with being seen as more "snappy". Any other ideas? WDYT? J. -- +48 660 796 129 =============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ===============================================================================
