potiuk opened a new issue, #43304: URL: https://github.com/apache/airflow/issues/43304
Currently all providers after #42505 are all in a single "providers" project. This has gone through several teething problems (mainly connected to bugs in `uv` and ways how to integrate development environment with IDEs such as Pycharm and VSCode, but seems that those problems are largely solved now and we can possibly move to the next step where each provider will have it's own `pyproject.toml` with its own dependencies and the workspace setting of `uv` will allow us to resolve all those dependencies together and keep our setup with constraints, CI image that is used for CI worfklow and breeze and local development of providers with Breeze. This has a number of changes to be implemented. Ideally each provider will have it's own complete "directory" where things are kept together: * code * docs * tests * system tests * dependencies defined in pyproject.toml * ability to build provider package straight using PEP-compliant frontends without having to copy providers code and generating some of the files (__init__.py, pyproject.toml, READMEs etc.) The package building is currently done dynamically via `breeze` commands, where code is extracted and pacakge is prepared, also dependency information (including devel dependencies) is kept in `provider.yaml`. Ideally all the information that is needed to generate dependencies and build packages, should be moved to `pyproject.toml` and our breeze/CI automation should retrieve information from there, rather than provider.yaml. This can be done in stages: 1) we could only move code and tests first - no docs or other files 2) we could do it provider-by-provider if we temporarily implement incremental change in our tooling to support both cases Or it could be done via automated script that would convert all providers at once - this was earlier POC's implementing this approach (not nearly close to be complete - just testing viability of such approach) that could be used as base for the new solution.: Script: https://github.com/apache/airflow/pull/28291 Result of runningn the script: https://github.com/apache/airflow/pull/28292 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
