jdu opened a new issue, #26930: URL: https://github.com/apache/airflow/issues/26930
### Description Because of the complexity of some of the plugins that are starting to come out, i'd like to propose a long-term goal / feature of finding a way to isolate plugins to prevent dependency issues, and stabilize integration points and problems over the longer term. We're starting to run into issues where plugins and extensions to airflow have really complex dependency trees which cause upgrades and installs to become much more complex as well as major versions can break previously working plugins triggering rewrites as well as version checking to decide which interface to work against. I know i'm light on details, but doing something similar to other tools where plugins are isolated in a separate process / environment, like LSP implementations in an editor, or how Sublime handles python-based extensions (I think they run as a background service when needed), where they are spawned as a secondary process and integrate with airflow through gRPC or something similar. ### Use case/motivation This would make developing plugins and extensions easier as there can be a contractual interface that developers work against that can insulate them from changes in airflows code and classes. We're right now dealing with an issue in upgrading to 2.4.x where a plugin we rely on heavily has been a bit of a horror show on 2.4.x a) Because it's own dependencies now conflict with airflows and some provider packages b) Because the interface it worked against changed (albeit the interface it worked against was probably not the correct one in the first place). ### Related issues _No response_ ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
