potiuk commented on issue #37484: URL: https://github.com/apache/airflow/issues/37484#issuecomment-1959188687
Yeah. You are completely right that GIL is an extra guard here as well. Even if you use Threads, in Python, GIL will absolulely prevent two parallel threads to run C functions like getenv/setenv to interfere with each other. This is also why we very rarely use threads in Python in general (and in Airflow) because they are not very useful for parallelism - except the situation where most of the threads are waiting for I/O operations. I think in C/Java those concerns about thread safety are valid, but in Python they are very, very unlikely to cause problems and it's pretty common pattern to pass things around through env variables in Python. This **migh** change with https://peps.python.org/pep-0703/ - when we will be able to have Python run with --no-gil option, but there are likely many years until it becomes a usable option at all and then we would have to probably adapt airlfow in various ways to make good use of it, so I would not complicate the code for that possibilty **just yet**. There will be likely (if we do it) a bigger effort involved requiring re-architecturing a number of things in Airflow when it happens - including changing our decisions how we run our tasks (for example I think it is more likely that we change the way how we fork/spawn into using sub-interpreters https://realpython.com/python312-subinterpreters/ way faster then exploring no-GIL. But that requires Python 3.12 support at the very least. And yeah. I think closing it makes sense for now :). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
