+1 for devel-common
On 04.03.25 20:21, Ferruzzi, Dennis wrote:
devel-common sounds reasonable
- ferruzzi
________________________________
From: Jarek Potiuk <ja...@potiuk.com>
Sent: Tuesday, March 4, 2025 10:53 AM
To: dev@airflow.apache.org
Subject: RE: [EXT] [DISCUSS] Turn "tests_common" into separate distribution for
development
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you can confirm the sender and know the
content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne
cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas
confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le
contenu ne présente aucun risque.
I am doing a bit more cleanup, and I have found that the easier way to fix some of the
remaining issues will be to clean-up (and remove) the remaining editable devel
dependencies and incorporate them all in the "tests-common" package.
You can take a look at the PR: https://github.com/apache/airflow/pull/47281 -
but basically what it means is:
* all "devel" dependencies are added as required dependencies of "tests-common" (except
"doc" - I will treat doc separately).
* I removed all "legacy" extras from the "airflow" package including "bundle" extras: "devel-ci",
"devel-db" and a few others - except installing "all" dependencies as they were pretty useless. Except "editable" all
only - see below - we will have no more devel and bundle extras (Ash - I guess this is what you were looking forward to :) )
* Instead we have one "all" extra that is available only in editable mode - it's not
documented in user documentation and it is really only useful to install everything with `pip`
(with uv you get the same with `uv sync --all-extras`) - this is still used internally in the CI
image to run `uv pip install .[all] --constraints` until we switch to use "uv.lock" in
the future
* hatch_build.py is significantly simpler and easier to understand now - with
all the bundle removal and moving all dependencies to ./tests-common
* we still need dynamic dependencies and ./hatch_build.py - but less and less,
with PEP735 (https://peps.python.org/pep-0735/) implemented in pip in April we
will likely be able to turn our optional dependencies into static
pyproject.toml deps, and with https://peps.python.org/pep-0771/ (needs approval
and implementation) we will likely be able to have static pyproject.toml
required dependencies as well.
* I updated install and contributing docs to be "uv first" - presenting as
recommended and the first option to go with `uv` - as it is becoming deceptively simple
now to work with both - airflow and providers (and it will be even simpler after few next
PRs)
Now. THE BIG QUESTION - naming again. With all those changes.....
`tests-common` is becoming more of a `devel-common` package - because what it
will do - it will contribute to all other sub-projects all the development
tooling that is needed for those other projects to be developed.
Shall we name it "devel-common" instead of `tests-common`?
Part of why I think it makes sense is this specification in `tests-common/pyproject.toml`
(see attachment) - also https://ibb.co/1cwZQRm if you do not see attachment. Note that
these are generally "common" devel dependencies - and each of the packages can
contribute their own (including task-sdk, airflow-core (future), every provider etc.)
[Screenshot from 2025-03-04 19-50-32.png]
J.
On Mon, Mar 3, 2025 at 4:14 AM Jarek Potiuk
<ja...@potiuk.com<mailto:ja...@potiuk.com>> wrote:
Hello everyone.
I created the PR for that https://github.com/apache/airflow/pull/47281.
It's even nicer than I anticipated. I love the new super-simple workflows this
restructuring finally enabled.
With `uv` and workspace, and the new structure of tests, developing and running
tests for providers, task-sdk or any of the other future sub-projects becomes
very, very straightforward, we avoid duplication of pytest options and
switching between airflow, task-sdk, providers tests will be simple and
straightforward.
1) First of all with this change, we remove `devel-tests` extra. It was always
needed in local venv to install all test dependencies, But this is completely
gone right now. Test dependencies are automatically installed now when you run
`uv sync` - and you do not need to specify `--extra devel-tests`. If you are
still using `pip` (I strongly recommend switching to uv) - you just install
`pip install -e ./tests-common`.
2) the previous way of syncing and running tests in "everything installed" mode
works as it worked before:
uv sync --all-extras
This installs all possible extras of airflow, allows you to run tests for all
providers, activate the venv in `.venv` and run `pytests tests/always` or
`pytest providers/mongo/tests` or `pytest task_sdk/tests and it should all
work fine as it did before
3) you can also install dependencies of a selected provider (or a few of those)
in your .venv
uv sync --extra mongo
3) All the `breeze testing provider-tests --test-type "Providers[mongo]"" and
generally all the ways to replicate what we run in CI inside breeze containers with selective
checks will also continue to work - no changes here.
But then, a little bit of standardization, a bit of pre-commits that
automatically update `pyproject.toml` files of ours with cross-provider
dependencies and a little of `uv` workspace magic sprinkled over the repo of
ours and now we can do something much more straightforward:
You can change directory to task_sdk, or provider folder of your choice and
simply run:
cd providers/mongo
uv run pytest
or
cd task_sdk
uv run pytest
And it will do what you would expect to do - even if this is the first thing
you do after checking out Airflow's repo without any earlier syncing or
preparation. Yes. It's that simple.
The `uv run pytest` command will now automatically: install python, create a
venv, sync your venv to ONLY have dependencies needed for your provider or any
of the providers imported by your provider, and all test dependencies needed,
it will find only non-system, non-integration tests and run them. That's it. If
you want to only develop one provider - you will not have to worry any more
about breeze image or setting up all other dependencies.
Go ahead and try it. It's unbelievably simple.
J.
On Tue, Feb 25, 2025 at 5:45 PM Ash Berlin-Taylor
<a...@apache.org<mailto:a...@apache.org>> wrote:
I like this approach, lets do it!
On 25 Feb 2025, at 16:02, Jarek Potiuk
<ja...@potiuk.com<mailto:ja...@potiuk.com>> wrote:
airflow-core
task-sdk
tests-common
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org
