Hello here, I wanted to draw a bit of attention to the PR I've been brewing this week (I had quite a bit less time this week due to my friend's visit but I am finally catching up :).
https://github.com/apache/airflow/pull/46608 I finally got it green after quite some iterations- and as explained to Jens it's rather difficult to split the "beast" - because it removes the "providers" umbrella package completely and moves the remaining pieces to have basic tests, integration and system tests to "inside" each provider "sub-project". This is not yet the final one, we will still have a number of follow-ups, to get to the "final" setup, but this one is moving us very, very close. I wanted to provide a bit of context, and (in parallel to merging this one). I wanted to discuss some follow up steps. The final goal is to be able to work on each provider's project completely separately from airflow "scheduler/task_sdk" and other packages. Ideally someone who wants to contribute to a provider should be able to just work on the provider's project - and not even care about airflow. But all that will be possible while keeping the "all together" workflow possible - where we will still contribute to a single repo, regardless if it is Airflow or provider, and where we can easily do refactorings and bulk changes that involve multiple providers easily. It's a bit of "Holy Grail" of the dev env I've been working towards for several years - and it seems that with the modern tooling and packaging standards we can "eat cake and have it too" finally. I still have some long term decisions to make about the final structure (especially how tests folders in ach provider should be named and what to do with "tests_common" project) - but I would love to have a separate discussion after the first step of cleanup and separation is done, and after I remove a few other pieces of dead code we have - around provider's manager - but also allowing the next steps where we switch providers and their tests one-by-one to support db-less mode, switching to task-sdk. So I sincerely apologise for the "beast" PR to the reviewers, but I also think it's worth spending a bit of time to get it reviewed and merged, to do all the follow up steps. If there is anything I can help with for the fellow reviewers, I am happy to. J.
