The PR is now merged,,, we have 'devel-common` now. The important bit now for everyone... It's easier than ever to have dev-env running for airflow...
1) uv sync -> and your .venv env allows you to run all tests for airflow/task-sdk 2) uv sync --all-packages -> and your .venv allows you to also run tests for all providers 3) cd providers/PROVIDER uv sync gives env ready to run all tests for the provider of your choice uv run pytest does it. There are likely a few missing per-provider deps but we will find and squash them soon ! Have fun! J. On Wed, Mar 5, 2025 at 12:48 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Seems like the PR is getting to "green" zone - so one last push - and I am > changing it to "devel-common" unless I hear strong NOOOOOOO! > > On Tue, Mar 4, 2025 at 10:36 PM Vincent Beck <vincb...@apache.org> wrote: > >> devel-common makes sense to me >> >> On 2025/03/04 21:13:47 "Oliveira, Niko" wrote: >> > +1 to devel-common from me >> > >> > ________________________________ >> > From: Ferruzzi, Dennis <ferru...@amazon.com.INVALID> >> > Sent: Tuesday, March 4, 2025 11:21:20 AM >> > To: dev@airflow.apache.org >> > Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate >> distribution for development >> > >> > CAUTION: This email originated from outside of the organization. Do not >> click links or open attachments unless you can confirm the sender and know >> the content is safe. >> > >> > >> > >> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur >> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous >> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas >> certain que le contenu ne présente aucun risque. >> > >> > >> > >> > devel-common sounds reasonable >> > >> > >> > - ferruzzi >> > >> > >> > ________________________________ >> > From: Jarek Potiuk <ja...@potiuk.com> >> > Sent: Tuesday, March 4, 2025 10:53 AM >> > To: dev@airflow.apache.org >> > Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate >> distribution for development >> > >> > >> > CAUTION: This email originated from outside of the organization. Do not >> click links or open attachments unless you can confirm the sender and know >> the content is safe. >> > >> > >> > >> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur >> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous >> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas >> certain que le contenu ne présente aucun risque. >> > >> > >> > I am doing a bit more cleanup, and I have found that the easier way to >> fix some of the remaining issues will be to clean-up (and remove) the >> remaining editable devel dependencies and incorporate them all in the >> "tests-common" package. >> > >> > You can take a look at the PR: >> https://github.com/apache/airflow/pull/47281 - but basically what it >> means is: >> > >> > * all "devel" dependencies are added as required dependencies of >> "tests-common" (except "doc" - I will treat doc separately). >> > * I removed all "legacy" extras from the "airflow" package including >> "bundle" extras: "devel-ci", "devel-db" and a few others - except >> installing "all" dependencies as they were pretty useless. Except >> "editable" all only - see below - we will have no more devel and bundle >> extras (Ash - I guess this is what you were looking forward to :) ) >> > * Instead we have one "all" extra that is available only in editable >> mode - it's not documented in user documentation and it is really only >> useful to install everything with `pip` (with uv you get the same with `uv >> sync --all-extras`) - this is still used internally in the CI image to run >> `uv pip install .[all] --constraints` until we switch to use "uv.lock" in >> the future >> > * hatch_build.py is significantly simpler and easier to understand now >> - with all the bundle removal and moving all dependencies to ./tests-common >> > * we still need dynamic dependencies and ./hatch_build.py - but less >> and less, with PEP735 (https://peps.python.org/pep-0735/) implemented in >> pip in April we will likely be able to turn our optional dependencies into >> static pyproject.toml deps, and with https://peps.python.org/pep-0771/ >> (needs approval and implementation) we will likely be able to have static >> pyproject.toml required dependencies as well. >> > * I updated install and contributing docs to be "uv first" - presenting >> as recommended and the first option to go with `uv` - as it is becoming >> deceptively simple now to work with both - airflow and providers (and it >> will be even simpler after few next PRs) >> > >> > Now. THE BIG QUESTION - naming again. With all those changes..... >> `tests-common` is becoming more of a `devel-common` package - because what >> it will do - it will contribute to all other sub-projects all the >> development tooling that is needed for those other projects to be developed. >> > >> > Shall we name it "devel-common" instead of `tests-common`? >> > >> > Part of why I think it makes sense is this specification in >> `tests-common/pyproject.toml` (see attachment) - also >> https://ibb.co/1cwZQRm if you do not see attachment. Note that these are >> generally "common" devel dependencies - and each of the packages can >> contribute their own (including task-sdk, airflow-core (future), every >> provider etc.) >> > >> > [Screenshot from 2025-03-04 19-50-32.png] >> > >> > >> > J. >> > >> > >> > >> > On Mon, Mar 3, 2025 at 4:14 AM Jarek Potiuk <ja...@potiuk.com<mailto: >> ja...@potiuk.com>> wrote: >> > Hello everyone. >> > >> > I created the PR for that https://github.com/apache/airflow/pull/47281. >> > >> > It's even nicer than I anticipated. I love the new super-simple >> workflows this restructuring finally enabled. >> > >> > With `uv` and workspace, and the new structure of tests, developing and >> running tests for providers, task-sdk or any of the other future >> sub-projects becomes very, very straightforward, we avoid duplication of >> pytest options and switching between airflow, task-sdk, providers tests >> will be simple and straightforward. >> > >> > 1) First of all with this change, we remove `devel-tests` extra. It was >> always needed in local venv to install all test dependencies, But this is >> completely gone right now. Test dependencies are automatically installed >> now when you run `uv sync` - and you do not need to specify `--extra >> devel-tests`. If you are still using `pip` (I strongly recommend switching >> to uv) - you just install `pip install -e ./tests-common`. >> > >> > 2) the previous way of syncing and running tests in "everything >> installed" mode works as it worked before: >> > >> > uv sync --all-extras >> > >> > This installs all possible extras of airflow, allows you to run tests >> for all providers, activate the venv in `.venv` and run `pytests >> tests/always` or `pytest providers/mongo/tests` or `pytest task_sdk/tests >> and it should all work fine as it did before >> > >> > 3) you can also install dependencies of a selected provider (or a few >> of those) in your .venv >> > >> > uv sync --extra mongo >> > >> > 3) All the `breeze testing provider-tests --test-type >> "Providers[mongo]"" and generally all the ways to replicate what we run in >> CI inside breeze containers with selective checks will also continue to >> work - no changes here. >> > >> > But then, a little bit of standardization, a bit of pre-commits that >> automatically update `pyproject.toml` files of ours with cross-provider >> dependencies and a little of `uv` workspace magic sprinkled over the repo >> of ours and now we can do something much more straightforward: >> > >> > You can change directory to task_sdk, or provider folder of your choice >> and simply run: >> > >> > cd providers/mongo >> > uv run pytest >> > >> > or >> > >> > cd task_sdk >> > uv run pytest >> > >> > And it will do what you would expect to do - even if this is the first >> thing you do after checking out Airflow's repo without any earlier syncing >> or preparation. Yes. It's that simple. >> > >> > The `uv run pytest` command will now automatically: install python, >> create a venv, sync your venv to ONLY have dependencies needed for your >> provider or any of the providers imported by your provider, and all test >> dependencies needed, it will find only non-system, non-integration tests >> and run them. That's it. If you want to only develop one provider - you >> will not have to worry any more about breeze image or setting up all other >> dependencies. >> > >> > Go ahead and try it. It's unbelievably simple. >> > >> > J. >> > >> > >> > >> > On Tue, Feb 25, 2025 at 5:45 PM Ash Berlin-Taylor <a...@apache.org >> <mailto:a...@apache.org>> wrote: >> > I like this approach, lets do it! >> > >> > > On 25 Feb 2025, at 16:02, Jarek Potiuk <ja...@potiuk.com<mailto: >> ja...@potiuk.com>> wrote: >> > > >> > > airflow-core >> > > task-sdk >> > > tests-common >> > >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> For additional commands, e-mail: dev-h...@airflow.apache.org >> >>
