I am doing a bit more cleanup, and I have found that the easier way to fix some of the remaining issues will be to clean-up (and remove) the remaining editable devel dependencies and incorporate them all in the "tests-common" package.
You can take a look at the PR: https://github.com/apache/airflow/pull/47281 - but basically what it means is: * all "devel" dependencies are added as required dependencies of "tests-common" (except "doc" - I will treat doc separately). * I removed all "legacy" extras from the "airflow" package including "bundle" extras: "devel-ci", "devel-db" and a few others - except installing "all" dependencies as they were pretty useless. Except "editable" all only - see below - we will have no more devel and bundle extras (Ash - I guess this is what you were looking forward to :) ) * Instead we have one "all" extra that is available only in editable mode - it's not documented in user documentation and it is really only useful to install everything with `pip` (with uv you get the same with `uv sync --all-extras`) - this is still used internally in the CI image to run `uv pip install .[all] --constraints` until we switch to use "uv.lock" in the future * hatch_build.py is significantly simpler and easier to understand now - with all the bundle removal and moving all dependencies to ./tests-common * we still need dynamic dependencies and ./hatch_build.py - but less and less, with PEP735 (https://peps.python.org/pep-0735/) implemented in pip in April we will likely be able to turn our optional dependencies into static pyproject.toml deps, and with https://peps.python.org/pep-0771/ (needs approval and implementation) we will likely be able to have static pyproject.toml required dependencies as well. * I updated install and contributing docs to be "uv first" - presenting as recommended and the first option to go with `uv` - as it is becoming deceptively simple now to work with both - airflow and providers (and it will be even simpler after few next PRs) Now. THE BIG QUESTION - naming again. With all those changes..... `tests-common` is becoming more of a `devel-common` package - because what it will do - it will contribute to all other sub-projects all the development tooling that is needed for those other projects to be developed. *Shall we name it "devel-common" instead of `tests-common`? * Part of why I think it makes sense is this specification in `tests-common/pyproject.toml` (see attachment) - also https://ibb.co/1cwZQRm if you do not see attachment. Note that these are generally "common" devel dependencies - and each of the packages can contribute their own (including task-sdk, airflow-core (future), every provider etc.) [image: Screenshot from 2025-03-04 19-50-32.png] J. On Mon, Mar 3, 2025 at 4:14 AM Jarek Potiuk <ja...@potiuk.com> wrote: > Hello everyone. > > I created the PR for that https://github.com/apache/airflow/pull/47281. > > It's even nicer than I anticipated. I love the new super-simple workflows > this restructuring finally enabled. > > With `uv` and workspace, and the new structure of tests, developing and > running tests for providers, task-sdk or any of the other future > sub-projects becomes very, very straightforward, we avoid duplication of > pytest options and switching between airflow, task-sdk, providers tests > will be simple and straightforward. > > 1) First of all with this change, we remove `devel-tests` extra. It was > always needed in local venv to install all test dependencies, But this is > completely gone right now. Test dependencies are automatically installed > now when you run `uv sync` - and you do not need to specify `--extra > devel-tests`. If you are still using `pip` (I strongly recommend switching > to uv) - you just install `pip install -e ./tests-common`. > > 2) the previous way of syncing and running tests in "everything installed" > mode works as it worked before: > > uv sync --all-extras > > This installs all possible extras of airflow, allows you to run tests for > all providers, activate the venv in `.venv` and run `pytests tests/always` > or `pytest providers/mongo/tests` or `pytest task_sdk/tests and it should > all work fine as it did before > > 3) you can also install dependencies of a selected provider (or a few of > those) in your .venv > > uv sync --extra mongo > > 3) All the `breeze testing provider-tests --test-type "Providers[mongo]"" > and generally all the ways to replicate what we run in CI inside breeze > containers with selective checks will also continue to work - no changes > here. > > But then, a little bit of standardization, a bit of pre-commits that > automatically update `pyproject.toml` files of ours with cross-provider > dependencies and a little of `uv` workspace magic sprinkled over the repo > of ours and now we can do something much more straightforward: > > *You can change directory to task_sdk, or provider folder of your choice > and simply run:* > > *cd providers/mongo* > *uv run pytest* > > or > > *cd task_sdk* > *uv run pytest * > > And it will do what you would expect to do - even if this is the first > thing you do after checking out Airflow's repo without any earlier syncing > or preparation. Yes. It's that simple. > > The `uv run pytest` command will now automatically: install python, create > a venv, sync your venv to ONLY have dependencies needed for your provider > or any of the providers imported by your provider, and all test > dependencies needed, it will find only non-system, non-integration tests > and run them. That's it. If you want to only develop one provider - you > will not have to worry any more about breeze image or setting up all other > dependencies. > > Go ahead and try it. It's unbelievably simple. > > J. > > > > On Tue, Feb 25, 2025 at 5:45 PM Ash Berlin-Taylor <a...@apache.org> wrote: > >> I like this approach, lets do it! >> >> > On 25 Feb 2025, at 16:02, Jarek Potiuk <ja...@potiuk.com> wrote: >> > >> > airflow-core >> > task-sdk >> > tests-common >> >>
