Hello here,
There was - apparently - one outstanding issue in the new provider
structure - VSCode had trouble with recognising imports for cross-provider
imports - because apparently it did not recognise implicit - also called
native - packages (i.e. folders without __init__.py). We tested it with
Mikhail and Jens and tried to get it working and I **think** we finally
succeeded to make it works for all cases (AKA "we have cake and ate it
too"):
* uv sync works
* pip install -e . -e ./providers/PROVIDER works
* pytest tests all work with adding the right PYTHONPATH
* Intellij properly recognizes the imports for both providers and
cross-providers
* VSCode also recognizes the imports and allows to navigate
You will still need to add "tests" folder as "test source" in IntelliJ (you
can also add "src" folders as sources but that should not be needed if you
use uv sync-ed virtualenv) if tests are cross-importing stuff from other
test files. This is documented in our contributing guide [1[ and Mikhail
will add equivalent screenshots for VScode
That required us to add many more `__init__.py` files with so-called
"legacy" namespaces (for airflow/providers/common etc. packages),. One
watchout - despite the "legacy" name - this form is not deprecated, it has
no impact on provider packages generated (the .whl files still have
implicit - essentially empty - namespace packages) and we can continue
building and releasing provider packages as usual. PR for that in [2] and
you can read more details there.
Maybe someday the support in IDES will be better and we might turn airflow
into fully "implicit packages driven" when we restructure airflow and we
will be able to remove all the unneeded __init__.py files then. But it is
not that day yet.
For now - we have a pre-commit that automatically generates and updates
files in providers ('check-providers-subpackages-init-file-exist') - so
you should not have to worry about doing anything when you add a new
provider the __init__.py files will be sorted for you automatically when
you commit your code. Just commit what it creates/updates.
You can read more about namespace packages in [3] - but this is a pretty
convoluted subject and we had to figure out how to do it best with a little
trial-and-error - especially because of various support from IDEs and other
tooling (mypy was particularly picky to get right with the new setup).
[1] Pycharm quick-start contributing guide:
https://github.com/apache/airflow/blob/main/contributing-docs/quick-start-ide/contributors_quick_start_pycharm.rst
[2] Legacy namespace PR: https://github.com/apache/airflow/pull/47064
[3] Namespace packages standard:
https://packaging.python.org/en/latest/guides/packaging-namespace-packages/
J.
