Overall +1 on this one. Regarding the naming, why not keeping "tests_common" instead of "common_test_code"? I am not a big fan of "common_test_code" but it is obviously just a personal opinion (as it is always with naming :))
On 2025/02/16 13:30:09 Jarek Potiuk wrote: > > Just wondernig... would an optional dependency not be the right place to > describe that apache-airflow-providers-google[tests] would have an > dependency to the common_tests subproject? > > Would mean you would need to install via > > pip install -e . -e ./task_sdk[tests] -e. ./providers/google[tests] > > Something like that. This is more of a details of workspace but we are > going to use "dev" dependency group for that - rather than extra. Details > to be worked out how pip interaction will look like (pip does not have > support for dependency groups yet - they are coming in 25.1 - they are > already merged in main) > > With `uv` dependency groups can be used today and "dev" dependency group is > installed automatically when you run uv sync or `uv pip install -e.` -> so > we will follow this along > > See details about "development dependencies" in uv here > https://docs.astral.sh/uv/concepts/projects/dependencies/#development-dependencies > > Dependency groups are already approved via PEP-735 > https://peps.python.org/pep-0735/ and as mentioned - we are a release away > from having them released in `pip`, so for now we will have to emulate it > with extras for pip case I think but we will be able to remove it when pip > 25.1 is released and matures a bit. > > J. > > > > > > > > > > On Sun, Feb 16, 2025 at 2:23 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > I would love to see some airflow_testing package which will be useful for > > testing airflow-related projects and involve independently. > > > > > Certainly, it's not a good thing to have tests import something from > > tests. > > New packages as projects are cheap and provide more flexibility and are > > useful from outside of the project. > > > > I already explained in the PR, but let me repeat my opinion here: > > > > IMHO It's extremely unlikely we are going to release and publish the > > common test code / fixtures in any way. They will continue to be in > > development-only-distribution and they will be treated as "internal detail". > > > > If we decide to release and publish them, we will have to maintain > > backwards compatibility and account for our users (like you) using them for > > their own purpose. That would block us or make it very difficult to make > > breaking changes in them. > > > > So while you will be free to continue copying the whole distribution and > > use it in your tests as you want (our licence allows that) - I seriously > > doubt we will ever release and publish it in "reusable form" with > > "compatibility guarantees". It's far more efficient if people like you just > > copy them and are aware that they can change any time. > > > > J. > > > > On Sun, Feb 16, 2025 at 2:21 PM Alexander Shorin <kxe...@apache.org> > > wrote: > > > >> I would love to see some airflow_testing package which will be useful for > >> testing airflow-related projects and involve independently. > >> > >> Certainly, it's not a good thing to have tests import something from > >> tests. > >> New packages as projects are cheap and provide more flexibility and are > >> useful from outside of the project. > >> > >> Also this project could have a future for testing, compatibility, quality > >> and rest of measuring. > >> > >> -- > >> ,,,^..^,,, > >> > >> > >> On Sun, Feb 16, 2025 at 4:12 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> > >> > Hello here, > >> > > >> > Next phase of the cleanup - it's been sped up by the comment from > >> @kxepal > >> > - https://github.com/apache/airflow/pull/46801#issuecomment-2661415731 > >> - > >> > I > >> > have planned to do it a bit later this week, but maybe indeed it's a > >> good > >> > idea to start a discussion now so that people are not confused. > >> > > >> > Currently we are using "from tests_common" to reuse code in various > >> > providers and airflow, and this is fine, with the exception that > >> > "tests_common" is currently just a package in "airflow" main project. > >> But > >> > it does not have to be (or rather it should not be). > >> > > >> > We should have it in a separate sub-project - similarly as providers/* > >> and > >> > task_sdk are now separate projects - and we should add dependency to the > >> > common test code distribution from within all the projects that use it > >> (via > >> > workspace feature). > >> > > >> > My proposal is to name it "common_test_code" (but I am open to any > >> > suggestions): It will look like this > >> > > >> > . > >> > |- airflow > >> > |- common_test_code > >> > | pyproject.toml > >> > | src > >> > | tests_common > >> > > >> > The code in airflow and providers will not change, they will continue to > >> > use "from tests_common import ...". > >> > > >> > * uv will work as it used to work, no changes will be needed - uv sync > >> will > >> > automatically install the common_test_code as editable project > >> > > >> > * additionally - (and that's a big plus) after this change you should be > >> > able to run `uv sync` in a provider folder and have provider-specific > >> > separate venv, and basically you should be able to run tests for a > >> provider > >> > using that .venv and generally treat provider project as a "standalone" > >> one > >> > - this is what I hinted at in the previous mail. > >> > > >> > * people who do not use uv but pip for example will have to manually add > >> > the `common_test_code` as an editable install in their venv. For > >> example: > >> > > >> > pip install -e . -e ./task_sdk -e. ./providers/google -e > >> ./common_test_code > >> > > >> > This is the next step in standardizing the layout of the Airflow project > >> > using workspaces. > >> > > >> > J. > >> > > >> > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
