potiuk commented on PR #58890: URL: https://github.com/apache/airflow/pull/58890#issuecomment-3599373034
I do not think we should invest in it. This feature will stop being needed pretty much completely after the full isolation of `task-sdk` from `airflow-core`. Once this is completed, the only modules that Dags will be able to load from `airflow.` will be "airflow.sdk` - which in essence can all be always pre-imported before in DagFileProcessorManager. Even if we will be loading some "airflow." stuff - this will be actually loaded from deprecated redirections to "airflow.sdk". I would say it's more worthwile to introduce a completely new feature where you can indeed specify a string list of which modules should be preloaded in DagFileProcessor. But it should be new config, new behaviour, new feature. Actually it would also be great if it can be paired with the COW improvement. All this pre-loading of specified "heavy" modules could be done at the very beginning of of DagFileProcessor start with the ["gc" dance](https://docs.python.org/3/library/gc.html#gc.freeze) to load all the specified modules and freeze()/unfreeze() every time ne DagFileProcessor parser is started. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
