Seems like the PR is getting to "green" zone - so one last push - and I am changing it to "devel-common" unless I hear strong NOOOOOOO!
On Tue, Mar 4, 2025 at 10:36 PM Vincent Beck <vincb...@apache.org> wrote: > devel-common makes sense to me > > On 2025/03/04 21:13:47 "Oliveira, Niko" wrote: > > +1 to devel-common from me > > > > ________________________________ > > From: Ferruzzi, Dennis <ferru...@amazon.com.INVALID> > > Sent: Tuesday, March 4, 2025 11:21:20 AM > > To: dev@airflow.apache.org > > Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate > distribution for development > > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > certain que le contenu ne présente aucun risque. > > > > > > > > devel-common sounds reasonable > > > > > > - ferruzzi > > > > > > ________________________________ > > From: Jarek Potiuk <ja...@potiuk.com> > > Sent: Tuesday, March 4, 2025 10:53 AM > > To: dev@airflow.apache.org > > Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate > distribution for development > > > > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > certain que le contenu ne présente aucun risque. > > > > > > I am doing a bit more cleanup, and I have found that the easier way to > fix some of the remaining issues will be to clean-up (and remove) the > remaining editable devel dependencies and incorporate them all in the > "tests-common" package. > > > > You can take a look at the PR: > https://github.com/apache/airflow/pull/47281 - but basically what it > means is: > > > > * all "devel" dependencies are added as required dependencies of > "tests-common" (except "doc" - I will treat doc separately). > > * I removed all "legacy" extras from the "airflow" package including > "bundle" extras: "devel-ci", "devel-db" and a few others - except > installing "all" dependencies as they were pretty useless. Except > "editable" all only - see below - we will have no more devel and bundle > extras (Ash - I guess this is what you were looking forward to :) ) > > * Instead we have one "all" extra that is available only in editable > mode - it's not documented in user documentation and it is really only > useful to install everything with `pip` (with uv you get the same with `uv > sync --all-extras`) - this is still used internally in the CI image to run > `uv pip install .[all] --constraints` until we switch to use "uv.lock" in > the future > > * hatch_build.py is significantly simpler and easier to understand now - > with all the bundle removal and moving all dependencies to ./tests-common > > * we still need dynamic dependencies and ./hatch_build.py - but less and > less, with PEP735 (https://peps.python.org/pep-0735/) implemented in pip > in April we will likely be able to turn our optional dependencies into > static pyproject.toml deps, and with https://peps.python.org/pep-0771/ > (needs approval and implementation) we will likely be able to have static > pyproject.toml required dependencies as well. > > * I updated install and contributing docs to be "uv first" - presenting > as recommended and the first option to go with `uv` - as it is becoming > deceptively simple now to work with both - airflow and providers (and it > will be even simpler after few next PRs) > > > > Now. THE BIG QUESTION - naming again. With all those changes..... > `tests-common` is becoming more of a `devel-common` package - because what > it will do - it will contribute to all other sub-projects all the > development tooling that is needed for those other projects to be developed. > > > > Shall we name it "devel-common" instead of `tests-common`? > > > > Part of why I think it makes sense is this specification in > `tests-common/pyproject.toml` (see attachment) - also > https://ibb.co/1cwZQRm if you do not see attachment. Note that these are > generally "common" devel dependencies - and each of the packages can > contribute their own (including task-sdk, airflow-core (future), every > provider etc.) > > > > [Screenshot from 2025-03-04 19-50-32.png] > > > > > > J. > > > > > > > > On Mon, Mar 3, 2025 at 4:14 AM Jarek Potiuk <ja...@potiuk.com<mailto: > ja...@potiuk.com>> wrote: > > Hello everyone. > > > > I created the PR for that https://github.com/apache/airflow/pull/47281. > > > > It's even nicer than I anticipated. I love the new super-simple > workflows this restructuring finally enabled. > > > > With `uv` and workspace, and the new structure of tests, developing and > running tests for providers, task-sdk or any of the other future > sub-projects becomes very, very straightforward, we avoid duplication of > pytest options and switching between airflow, task-sdk, providers tests > will be simple and straightforward. > > > > 1) First of all with this change, we remove `devel-tests` extra. It was > always needed in local venv to install all test dependencies, But this is > completely gone right now. Test dependencies are automatically installed > now when you run `uv sync` - and you do not need to specify `--extra > devel-tests`. If you are still using `pip` (I strongly recommend switching > to uv) - you just install `pip install -e ./tests-common`. > > > > 2) the previous way of syncing and running tests in "everything > installed" mode works as it worked before: > > > > uv sync --all-extras > > > > This installs all possible extras of airflow, allows you to run tests > for all providers, activate the venv in `.venv` and run `pytests > tests/always` or `pytest providers/mongo/tests` or `pytest task_sdk/tests > and it should all work fine as it did before > > > > 3) you can also install dependencies of a selected provider (or a few of > those) in your .venv > > > > uv sync --extra mongo > > > > 3) All the `breeze testing provider-tests --test-type > "Providers[mongo]"" and generally all the ways to replicate what we run in > CI inside breeze containers with selective checks will also continue to > work - no changes here. > > > > But then, a little bit of standardization, a bit of pre-commits that > automatically update `pyproject.toml` files of ours with cross-provider > dependencies and a little of `uv` workspace magic sprinkled over the repo > of ours and now we can do something much more straightforward: > > > > You can change directory to task_sdk, or provider folder of your choice > and simply run: > > > > cd providers/mongo > > uv run pytest > > > > or > > > > cd task_sdk > > uv run pytest > > > > And it will do what you would expect to do - even if this is the first > thing you do after checking out Airflow's repo without any earlier syncing > or preparation. Yes. It's that simple. > > > > The `uv run pytest` command will now automatically: install python, > create a venv, sync your venv to ONLY have dependencies needed for your > provider or any of the providers imported by your provider, and all test > dependencies needed, it will find only non-system, non-integration tests > and run them. That's it. If you want to only develop one provider - you > will not have to worry any more about breeze image or setting up all other > dependencies. > > > > Go ahead and try it. It's unbelievably simple. > > > > J. > > > > > > > > On Tue, Feb 25, 2025 at 5:45 PM Ash Berlin-Taylor <a...@apache.org > <mailto:a...@apache.org>> wrote: > > I like this approach, lets do it! > > > > > On 25 Feb 2025, at 16:02, Jarek Potiuk <ja...@potiuk.com<mailto: > ja...@potiuk.com>> wrote: > > > > > > airflow-core > > > task-sdk > > > tests-common > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
