potiuk commented on issue #47608: URL: https://github.com/apache/airflow/issues/47608#issuecomment-2727541575
> Yes, I believe there was an email thread regarding this started by [@potiuk](https://github.com/potiuk) Now.. I recall one thing. - the only problem with getting rid of sequential executor, is that it will potentially make it impossible in some cases to run "production like" airflow configuration on Mac OS (i.e. harming development environment for some people). The problem is that some people suffered with Local Executor doing forks, where sequential executor did not do forks. Which is potentially disruptive especially for people who are running on MacOS Example discussion here https://github.com/apache/airflow/discussions/44243 explaining the problem Discussion that I started was here : https://lists.apache.org/thread/cc7wfz9w8d1npopj72nlxnh52dpzbls9 I am not sure if we can do much for MacOS until we get rid of the potential initializations that are executed through `airflow.__init__` which I **believe** is the root of the problem. The main problem is that if any library starts any threads before first fork happens, this often leads to SEGV problems - and this is inherent problem of running Python on MacOS - that is not really solvable on a system level. See for example discussion here: https://discuss.python.org/t/concerns-regarding-deprecation-of-fork-with-alive-threads/33555 What I think triggers this problem is combination of factors: * macOS * configuration in local settings that imports some libraries that start threads (mostly for logging as logging is initlalized on airflow `__init__.py` effectively * the fact that we need to import `airflow.__init__.py` to do anything - including importing task-sdk package on execution and dag file processing side * and then we fork in case of sequential executor just before we start task - if some threads are started by any of the 3rd-party libraries that randomly leads to SIGSEGV The problem with it is that it's next to impossible to tell user what was the root cause - some import somewhere of some library ... I think we can deal with it in two ways: * rearchitect initial import to not do all the initialization on just importing airflow * maybe (not sure if possible) we could check if there are any started threads before forking and "FAIL" - asking user to change configuration of logging or look at their libraries.... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
