So maybe instead we should rather check whether sqlite and/or sequential executor have been configured in the airflow instance, for recognizing a dev environment.
Instead of failing hard, it would be really cool to show a hint (big, red, flashy, …) directly in the UI to the users, at least to those with admin-privs. we could remind them of them using the dev-setup. and then also have some info text that names the disadvantages of the dev-setup like being slow etc, and then hinting to an easy alternative like docker-compose that one /could/ use in production as well.
I am not sure if we have this atm, but pre-configuring airflow with sqlite+sequentialexecutor is key to getting new users using airflow in the first place. those with low infrastructure expertise would just be scared of right away, I fear. and anyone who just wants to give it a shot as well.
Best, Lars On 7/21/21 12:36 AM, Andrew Godwin wrote:
We explored a similar idea with Django many years ago, and the conclusion back then, which I would also put forward here, is that having a project scale down to an easy developer install is of crucial importance, and so I think SQLite has to stay in that role(as
there is no reasonable alternative, at least not yet).I do think it should be heavily discouraged in production installs, though. If there's a way you think we can pull that off while not making development annoying, I'd be all for it.AndrewOn Tue, Jul 20, 2021 at 12:24 PM Shaw, Damian P. <[email protected] <mailto:[email protected]>> wrote:Some thought as a user of Airflow, I wouldn’t of adopted Airflow in the first place if I couldn’t test it with sqlite. And would be the same today, accessing docker isn’t always an easy in some companies. But having a warning that sqlite is development only and much slower than other solutions when it’s enabled seems fair. Also forcing new time users to edit the config on first run I think is acceptable as they will need to get used to do that frequently anyway if they’re rolling their own install. Damian *From:*Jarek Potiuk <[email protected] <mailto:[email protected]>> *Sent:* Tuesday, July 20, 2021 12:23 *To:* [email protected] <mailto:[email protected]> *Subject:* [DISCUSSION] Should we be more explicit about SQLite using for dev only (or kill it for non-dev entirely????) Hello Community, Recently we had several people who complained (on slack) that airflow 2.1 is slow in scheduling tasks. After some discussion it usually turned out that those people were using SQLite + Sequential executor. I think it gives very bad impression to users. We even had one user who almost gave up Airflow seeing how slow it is in scheduling tasks (!). I think while in Airflow 1.10 the difference was not as noticeable, Airflow 2 with Postgres/MySQL is lightning fast comparing to sqlite. It's like a different world. First time users might get a very bad impression when their first contact with Airflow is via sqlite + Sequential executor. Many people choose sqlite as their first choice when they try Airflow (Sqlite is generally seen as solid choice in many cases and people are afraid that setting up MySQL/Postgres might take them a lot of time to setup). However with current Docker-Compose quilck-start by Kamil it is already rather quick to set-up a working setup with Postgres. My idea is - why don't we make SQLITE "development-only" choice. That would require editable, development version of airflow to run and fail hard when it is installed as regular package (with appropriate "Use proper database - MySQL/Postgres" - and MSSQL when we release MSSQL-support in 2.2 ). I think that would be possible, it would not violate backwards compatibility (sqlite was anyhow for development-only) and it would help Airflow with being seen as more "snappy". Any other ideas? WDYT? J.--+48 660 796 129 ============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html <http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html> ==============================================================================
OpenPGP_signature
Description: OpenPGP digital signature
