potiuk commented on issue #6627: [AIRFLOW-5931] Use os.fork when appropriate to speed up task execution. URL: https://github.com/apache/airflow/pull/6627#issuecomment-557299451 But @dimberman -> multiprocessing also uses os.fork() underneath in fork mode (default for Linux). I have my reservations with using mutlprocessing (but mostly because people do not realise that it actually uses fork (and we plan to use it anyway so no difference). Using multiprocessing might be a more portable way if we consider running it in different environments. Note that in python 3.8 default mode for the new process is spawn as forking on MacOS might cause crashes because threads are not safe for forking and some system libraries on MacOS run threads. So using multiprocessing.Process will be slower on MacOS in 3.8 but won't crash.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
