GitHub user potiuk added a comment to the discussion: Can `with DAG(..)` be made thread local?
I don't think it makes sense to run dags in separate threads. This: ``` # time consuming operation ``` Is something that should never happen. If you read best practices in our dags it's explicitly **wrong** to run time consuming operation when dag is being parsed. Don't. You can do long running operations inside your "execute" methods of tasks, yes, but not when the Dags are parsed to establish the structure of the dag and let the Dag be serialized for scheduling. Also threads in Python do something different than you think they do most likely. At least until Python 3.14 there is GIL (global interpreter lock) and Threads do not not help with running python code in parallel (they run concurrently but not in paralllel) - it only helps in case you have I/O operation and waiting for them by several threads essentially (which again is strictly "no-no" when it comes to best practices of Dag parsing). So I see no particular reason **why** you would like to run parsing of Dags in separate threads as it gives you **nothing** if you follow good practice. Every Dag file is parsed in a separate process already to achieve paralllelism, so if you want to achieve parallelism, make several Dag files, import the common code from them and you are done. There is absolutely no need (and again - it makes no sense IMHO) to run Dag parsing in separate threads. GitHub link: https://github.com/apache/airflow/discussions/53691#discussioncomment-13873413 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
