Two small things (aka possible teething issues): * *uv self update* <- once you do it, in the future you will get a clear error when you should update your uv if you have not done so * *uv sync --reinstall* <- forces reinstallation of airflow package. Uv does it periodically with some heuristics, but this one will force it after restructuring
J. On Wed, Mar 5, 2025 at 11:13 PM Jarek Potiuk <ja...@potiuk.com> wrote: > The PR is now merged,,, we have 'devel-common` now. > > The important bit now for everyone... It's easier than ever to have > dev-env running for airflow... > > 1) uv sync -> and your .venv env allows you to run all tests for > airflow/task-sdk > 2) uv sync --all-packages -> and your .venv allows you to also run tests > for all providers > 3) > cd providers/PROVIDER > uv sync > > gives env ready to run all tests for the provider of your choice > > uv run pytest > > does it. > > There are likely a few missing per-provider deps but we will find and > squash them soon ! > > Have fun! > > J. > > > > On Wed, Mar 5, 2025 at 12:48 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> Seems like the PR is getting to "green" zone - so one last push - and I >> am changing it to "devel-common" unless I hear strong NOOOOOOO! >> >> On Tue, Mar 4, 2025 at 10:36 PM Vincent Beck <vincb...@apache.org> wrote: >> >>> devel-common makes sense to me >>> >>> On 2025/03/04 21:13:47 "Oliveira, Niko" wrote: >>> > +1 to devel-common from me >>> > >>> > ________________________________ >>> > From: Ferruzzi, Dennis <ferru...@amazon.com.INVALID> >>> > Sent: Tuesday, March 4, 2025 11:21:20 AM >>> > To: dev@airflow.apache.org >>> > Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate >>> distribution for development >>> > >>> > CAUTION: This email originated from outside of the organization. Do >>> not click links or open attachments unless you can confirm the sender and >>> know the content is safe. >>> > >>> > >>> > >>> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur >>> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous >>> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas >>> certain que le contenu ne présente aucun risque. >>> > >>> > >>> > >>> > devel-common sounds reasonable >>> > >>> > >>> > - ferruzzi >>> > >>> > >>> > ________________________________ >>> > From: Jarek Potiuk <ja...@potiuk.com> >>> > Sent: Tuesday, March 4, 2025 10:53 AM >>> > To: dev@airflow.apache.org >>> > Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate >>> distribution for development >>> > >>> > >>> > CAUTION: This email originated from outside of the organization. Do >>> not click links or open attachments unless you can confirm the sender and >>> know the content is safe. >>> > >>> > >>> > >>> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur >>> externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous >>> ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas >>> certain que le contenu ne présente aucun risque. >>> > >>> > >>> > I am doing a bit more cleanup, and I have found that the easier way to >>> fix some of the remaining issues will be to clean-up (and remove) the >>> remaining editable devel dependencies and incorporate them all in the >>> "tests-common" package. >>> > >>> > You can take a look at the PR: >>> https://github.com/apache/airflow/pull/47281 - but basically what it >>> means is: >>> > >>> > * all "devel" dependencies are added as required dependencies of >>> "tests-common" (except "doc" - I will treat doc separately). >>> > * I removed all "legacy" extras from the "airflow" package including >>> "bundle" extras: "devel-ci", "devel-db" and a few others - except >>> installing "all" dependencies as they were pretty useless. Except >>> "editable" all only - see below - we will have no more devel and bundle >>> extras (Ash - I guess this is what you were looking forward to :) ) >>> > * Instead we have one "all" extra that is available only in editable >>> mode - it's not documented in user documentation and it is really only >>> useful to install everything with `pip` (with uv you get the same with `uv >>> sync --all-extras`) - this is still used internally in the CI image to run >>> `uv pip install .[all] --constraints` until we switch to use "uv.lock" in >>> the future >>> > * hatch_build.py is significantly simpler and easier to understand now >>> - with all the bundle removal and moving all dependencies to ./tests-common >>> > * we still need dynamic dependencies and ./hatch_build.py - but less >>> and less, with PEP735 (https://peps.python.org/pep-0735/) implemented >>> in pip in April we will likely be able to turn our optional dependencies >>> into static pyproject.toml deps, and with >>> https://peps.python.org/pep-0771/ (needs approval and implementation) >>> we will likely be able to have static pyproject.toml required dependencies >>> as well. >>> > * I updated install and contributing docs to be "uv first" - >>> presenting as recommended and the first option to go with `uv` - as it is >>> becoming deceptively simple now to work with both - airflow and providers >>> (and it will be even simpler after few next PRs) >>> > >>> > Now. THE BIG QUESTION - naming again. With all those changes..... >>> `tests-common` is becoming more of a `devel-common` package - because what >>> it will do - it will contribute to all other sub-projects all the >>> development tooling that is needed for those other projects to be developed. >>> > >>> > Shall we name it "devel-common" instead of `tests-common`? >>> > >>> > Part of why I think it makes sense is this specification in >>> `tests-common/pyproject.toml` (see attachment) - also >>> https://ibb.co/1cwZQRm if you do not see attachment. Note that these >>> are generally "common" devel dependencies - and each of the packages can >>> contribute their own (including task-sdk, airflow-core (future), every >>> provider etc.) >>> > >>> > [Screenshot from 2025-03-04 19-50-32.png] >>> > >>> > >>> > J. >>> > >>> > >>> > >>> > On Mon, Mar 3, 2025 at 4:14 AM Jarek Potiuk <ja...@potiuk.com<mailto: >>> ja...@potiuk.com>> wrote: >>> > Hello everyone. >>> > >>> > I created the PR for that https://github.com/apache/airflow/pull/47281 >>> . >>> > >>> > It's even nicer than I anticipated. I love the new super-simple >>> workflows this restructuring finally enabled. >>> > >>> > With `uv` and workspace, and the new structure of tests, developing >>> and running tests for providers, task-sdk or any of the other future >>> sub-projects becomes very, very straightforward, we avoid duplication of >>> pytest options and switching between airflow, task-sdk, providers tests >>> will be simple and straightforward. >>> > >>> > 1) First of all with this change, we remove `devel-tests` extra. It >>> was always needed in local venv to install all test dependencies, But this >>> is completely gone right now. Test dependencies are automatically installed >>> now when you run `uv sync` - and you do not need to specify `--extra >>> devel-tests`. If you are still using `pip` (I strongly recommend switching >>> to uv) - you just install `pip install -e ./tests-common`. >>> > >>> > 2) the previous way of syncing and running tests in "everything >>> installed" mode works as it worked before: >>> > >>> > uv sync --all-extras >>> > >>> > This installs all possible extras of airflow, allows you to run tests >>> for all providers, activate the venv in `.venv` and run `pytests >>> tests/always` or `pytest providers/mongo/tests` or `pytest task_sdk/tests >>> and it should all work fine as it did before >>> > >>> > 3) you can also install dependencies of a selected provider (or a few >>> of those) in your .venv >>> > >>> > uv sync --extra mongo >>> > >>> > 3) All the `breeze testing provider-tests --test-type >>> "Providers[mongo]"" and generally all the ways to replicate what we run in >>> CI inside breeze containers with selective checks will also continue to >>> work - no changes here. >>> > >>> > But then, a little bit of standardization, a bit of pre-commits that >>> automatically update `pyproject.toml` files of ours with cross-provider >>> dependencies and a little of `uv` workspace magic sprinkled over the repo >>> of ours and now we can do something much more straightforward: >>> > >>> > You can change directory to task_sdk, or provider folder of your >>> choice and simply run: >>> > >>> > cd providers/mongo >>> > uv run pytest >>> > >>> > or >>> > >>> > cd task_sdk >>> > uv run pytest >>> > >>> > And it will do what you would expect to do - even if this is the first >>> thing you do after checking out Airflow's repo without any earlier syncing >>> or preparation. Yes. It's that simple. >>> > >>> > The `uv run pytest` command will now automatically: install python, >>> create a venv, sync your venv to ONLY have dependencies needed for your >>> provider or any of the providers imported by your provider, and all test >>> dependencies needed, it will find only non-system, non-integration tests >>> and run them. That's it. If you want to only develop one provider - you >>> will not have to worry any more about breeze image or setting up all other >>> dependencies. >>> > >>> > Go ahead and try it. It's unbelievably simple. >>> > >>> > J. >>> > >>> > >>> > >>> > On Tue, Feb 25, 2025 at 5:45 PM Ash Berlin-Taylor <a...@apache.org >>> <mailto:a...@apache.org>> wrote: >>> > I like this approach, lets do it! >>> > >>> > > On 25 Feb 2025, at 16:02, Jarek Potiuk <ja...@potiuk.com<mailto: >>> ja...@potiuk.com>> wrote: >>> > > >>> > > airflow-core >>> > > task-sdk >>> > > tests-common >>> > >>> > >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >>> For additional commands, e-mail: dev-h...@airflow.apache.org >>> >>>
