potiuk commented on PR #37908: URL: https://github.com/apache/airflow/pull/37908#issuecomment-1979983139
> What would be the huge impact? A single load-time check seems unlikely to cause any noticeable performance change. NOT forking when you can is the huge impact. Executing every single task takes few seconds more, because you need to spawn new interpreter, import 100s of modules and initialize them. DAG file processing has to do exactly the same (spawn new interpreter and import all that is needed to parse EVERY SINGLE DAG FILE - EVERY TIME IT DOES IT - i.e. every 30 seconds by default. Forks also use copy-on-write memory, which means that hen you for such pre-intialized processes, they use nearly 0 extra memory - even if you bave many of them, because most memory initialized in the parent process is not modified and shared between all the child processes. This is a huge performance hit. For example if you run your airlfow on your Mac it means that it will be almost constantly (due to DAG file processor) using a lot of CPU and memory for just scannind the DAG folder. A LOT more CPU and A LOT more CPU. Likely when you run airflow in your local virtualenv time of your battery in your laptop will be at least halved if not decreased by 60%. Not mentioning the environmental effect. It's not "one switch check", it's the fact that you will stop using forking on MacOS during development that is problematic. You have to be very, very careful, when you want to propose such a change, especially knowing that you will impact the development environment, ergonomy and quality of development for many of the contributiors - my "rough" assesment is that at least 50%-60% of our contributors use mac, and likely 80-90% of maintainers. You basically propose to decrease their quality-of-work significantly be **just** disabling forking on MacOS -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
