potiuk commented on issue #8674: URL: https://github.com/apache/airflow/issues/8674#issuecomment-628420469
Great analysis @jhtimmins thanks! Indeed - we discussed the fork/spawn case in the past and the MacOS vs. Linux change in 3.8. I think we are not at all concerned with Windows - anyhow the only recommended way to run Airflow on Windows is WSL2. Regarding MacOS vs. Linux. I think it's great it works on MacOS for development purposes, but for production, I believe we only ever support Linux. Also, we are moving more and more into Dockerised solutions - with official production image, coming docker-compose support, Helm Chart - we are Docker-first for execution of Airflow in production. For development we also have Breeze environment which I think long term might become the default and while we still support (and will support) running local virtualenv on MacOS, we already have a number of tests that will not work straight from the IDE or when run locally - they need integrations (rabbitmq, Cassandra, databases etc.) and for those Breeze's docker environment is the one that should be used. We have full MacOS support for Breeze as well so I think making sure tests pass and are fast on MacOS natively is not a huge problem. I think our main concern should be that all true "unit" tests work in any environment but the more complex "integration" tests can only work in Breeze. I think it really depends how many tests will not work. I believe this is a very small subset of tests. Those are the important ones, but I believe there are not many of those. If that's the case, then I personally would be for hard-coding forking and marking the tests that are bound to fail on MacOS with a custom pytest marker (say @pytest.marker.linux ). Those tests will then be automatically skipped when run on MacOS and the message we can put is "These tests only work in Linux environment, please use Breeze docker environment if you are running them on MacOS" or similar. This way we avoid people being surprised by failing tests, give them clear information how to run them. At the same time if people develop on Linux, they will still be able to run those tests "natively" without having to switch to breeze environment. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
