potiuk commented on PR #53390: URL: https://github.com/apache/airflow/pull/53390#issuecomment-3092329750
There is a potential problem I see here. On Celery worker when you have multiple tasks in parallel, potentially the same `__pycache__` folder is used by parallel workers running. I am not sure if we can do what you want to do without getting into some race conditions. Also I am not sure what problem it solves to be honest - the `__pycache__` is generally very well rebuilt when needed. I can think of one case where it can be problematic - when you are using same `__pycache__` folders for different Python versions, you will have some version code errors. But there, I'd say a better solution might be to set PYTHONDONOTWRITEBYTECODE before launching interpreter. Yes it **might** be slower a bit because parsing and bytecode generation happens every time, but in vast majority of cases when tasks runs for seconds, this will be negligible overhead - unless you have vast amount of code to parse. Could you please elaborate a bit more what exactly problem we are trying to solve here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
