FYI. Likely today I am going to "undraft" :)
https://github.com/apache/airflow/pull/47798.
*TL;DR; My proposal on the process of merging "airflow-core" move (it will
be slightly disturbing and require some local env adjustments by all
people). Read on - or leave it for Friday morning lecture :)*
I am trying to solve a few last riddles that Sphinx is asking me to solve,
getting PROD image in order and having to make sure that several
distributions produced are equivalent to the single distribution we had so
far when installed together. That all seems doable today. Most of the
tests, scripts, release process and tooling updates, development workflows,
breeze image building and tests are already fixed in the PR. I will also
add more detailed description what's "in" the PR - I had to make a number
of small changes on how we are importing stuff and where dependencies are
kept etc (hint - it's going to be much, much more standard) but those are
mostly impacting development environment, rather than released
distributions.
I will have a few follow-up things after that - but they will be far
smaller and far less - potentially - disruptive.
I keep on rebasing the change and while it's not **too** difficult, it
might be difficult for some people to rebase and resolve their conflicts,
also I expect a few teething issues - mostly around CI/ breeze, so I think
the following plan of making it happen is going to be the best:
1) I will make the PR green and hopefully reviewed by the end of tomorrow -
or generally by the time last beta is released (it's mostly renames and
fixing stuff to be used from new place - so even if it is a lot of
changes/files, it should be somewhat easy to review)
2) I will merge it **right after** the beta is released. This means that I
will merge and resolve conflicts for all PRs merged before beta3 (so hurry
up! :) )
3) After that - everyone will be asked to do four things:
a) rebase their opened PRs to latest main (and solve conflicts)
b) run `*breeze ci image build --python 3.9*" (for doc building and
some static checks)
c) run `*uv self upgrade*` followed by `*uv sync --reinstall*`
(possibly with `*--all-packages*` if they work on both - airflow core and
providers at the same time.
d) update their IDE settings - if they use IDEs - to make sure
airflow-core/src and airflow/core/tests are available for code completion
and import resolution (there will be updated screenshots on how to do it)
4) With that -> everything ***SHOULD*** work without big problems but I
will make myself available to solve any teething issues (and I am sure
other team members of the "CI/DEV task force" will help as well).
Also I have been somewhat out (speaking and taking part in a great L8 Conf
event in Warsaw Mon/Tue). I will be able to spend a lot of time Friday and
weekend to fix any potential teething issues - so that starting Monday next
week we can all continue working towards RC1 in an uninterrupted way.
I am also going to be very available to fix any potential issues next week
- while I will be completing some left-over tasks (I hope to get Sphinx a
little less mysterious by moving the docs to a proper "devel-common" place
for one and start identifying DB tests in providers and paving a path on
how we can get rid of those).
Let me know if you have any questions - I will do more docs and explanation
and also happy to talk on slack and explain more of the changes once the PR
is undrafted.
J.
On Fri, Mar 14, 2025 at 8:27 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hey here,
>
> I have a first (very draft and still requires a number of changes) PR for
> the final step of big refactoring of our projects and using workspace. This
> is to let you know about the changes coming (so please take a look at the
> consequences to not be surprised).
>
> This is the most *scary* one -> moving all airflow code to
> "airflow-core". And I have draft version of it in
> https://github.com/apache/airflow/pull/47798
>
> And it's not for the faint of heart :)
>
> [image: image.png]
>
> Note! It's not yet complete and unless you have some general comments,
> it's likely not worth pointing to individual changes (yet) - it's more to
> take a look at how things will look like eventually. I will work in the
> next two days to get it to reviewable state, and will keep it rebased and
> running till mid next-week. I would like to have it ready (including the
> release process) for the fourth (and final?) beta).
>
> Some resulting packaging changes:
>
> *FOR DEVELOPMENT:*
>
> * the pyproject.toml in the "root" of Airflow is still "apache-airflow"
> package - but this will be an empty "meta" package that will install
> together "apache-airflow-core", "apache-airflow-task-sdk" and optionally
> providers (via extras)
>
> * the airflow-core is a new "apache-airflow-core" distribution, where only
> airflow dependencies and airflow "core" extras are configured (smtp/ otel,
> pandas,rabbitmq etc) - I will likely cleanup some of those as well, some of
> them are not needed. the nice thing is that this package has all
> dependencies static (no hatch_build.py - everything is in pyproject.toml) -
> which is pretty cool and allow us to better use dependabot for security
> upgrades and notifications
>
> The airflow-core structure is pretty standard:
>
> airflow-core # <- this is folder where airflow-core distribution is
> \- src
> | \ airflow # <- This is airflow package
> | \- api
> | |- api_fastapi
> | |- assets
> | ...
> |- tests
> | \- always
> | |- api
> | ...
> |- docs
> |
> |- pyproject.toml
> |- README.md
>
>
> * for development - i will describe later the `pypi` way, but with `uv`
> things get simpler and we have a few new options (Dennis - this is
> continuation of discussion on the uv sync commands, so it's worth to
> look closely:
>
> There are a number of ways you will be (eventually able to interact with
> venv. After you checkout Airflow. You can change working directory and work
> on different packages and depending on which directory you run `uv sync` -
> uv (using workspace feature) will sync the **expected** dependencies.
>
> It's best to get used to the fact that instead of one airflow project we
> will have ~100 pretty independent projects, and while you can continue
> working with all of them as a single huge "workspace", it is generally way
> more convenient to change directory to the "distribution" you are working
> on currently and do everything there - with isolated set of dependencies
> required only for that "distribution" - "airflow-core", "task-sdk",
> "providers/amazon", "providers/mongo" - those are all separate
> distributions, and more and more we will be able to treat them as
> independent projects (but we will conveniently keep the option to develop
> and run tests in a joined "workspace" environment at the top of the project
> where we can install and test everything together - that's a bit of `uv
> workspace` magic in play.
>
> Here are typical patterns:
>
> 1) Installing all development dependencies for everything (I.e complete
> environment like in breeze) -- allows to run all tests for all airflow and
> all providers
>
> cd .
> uv sync --all-packages
>
> 2) installing just airflow core with required dependencies (ready for most
> core tests)
>
> cd airflow-core
> uv sync
>
> 3) installing airflow core with optional dependencies (should allow to run
> all core tests - including for the optional core features such as otel etc).
>
> cd airflow-core
> uv sync --all-extras
>
> 4) installing individual provider dependencies (say amazon) - this allows
> to run all tests of the provider you are working on - including installing
> all dependencies from cross-provider dependencies (i.e. if you have google
> tests in amazon provider, it will also install necessary google
> dependencies).
>
> cd providers/amazon
> uv sync
>
> Generally speaking - "airflow-core" will become (eventually) a truly
> airflow-only distribution. It will have a few dependencies to "standard"
> and "fab" providers - but I hope we will be able to get rid of those during
> the resulting cleanup.
>
> The IDE (IntelliJ) setting will just require "airflow-core/src" and
> "airflow-core/tests" to be source/test roots as usual for other
> distributions.
>
> I will update the docs after I complete the PR, there are some small
> variations on when to install which extras and I will play a bit to get to
> the best developer experience and least surprises.
>
> *FOR USERS*
>
> For "installable" airflow (i.e. user's experience) - the changes will be
> pretty much 100% transparent. When user will install "apache-airflow" or
> "apache-airflow[google]" - things will work as they did before - only
> instead of one "apache-airflow" distribution, they will have
> "apache-airflow", "apache-airflow-core" and "apache-airflow-task-sdk"
> installed.
>
> Regarding version numbers etc., I will start a separate discussion - later
> next week after we see how those packages will interact ("apache-airflow"
> will only contain extras, but for compatibility reasons we likely want to
> pin both "apache-airflow" and "apache-airflow-core" to each other, so that
> users will be able to upgrade "core" by upgrading "apache-airflow" - we do
> not want to change those habits likely.
>
> The "apache-airflow-task-sdk" will be versioned separately.
>
> Please take a look - also at the PR, see if you have any big
> issues/questions/doubts - let's start discussion here - I am happy to
> answer all general questions and adapt the PR to respond to
> questions/suggestions.
>
> In the meantime I will be working on making the PR green and adding
> missing bits and pieces for the release process.
>
> J.
>
>
>
>
