Woohoo, amazing news! Sounds great! Another huge structural change that is defining the `core` as reality, it won't be just definition anymore. This will enable lots of possibilities. Thanks Jarek!
On Wed, 19 Mar 2025, 14:52 Jarek Potiuk, <ja...@potiuk.com> wrote: > FYI. Likely today I am going to "undraft" :) > https://github.com/apache/airflow/pull/47798. > > *TL;DR; My proposal on the process of merging "airflow-core" move (it will > be slightly disturbing and require some local env adjustments by all > people). Read on - or leave it for Friday morning lecture :)* > > I am trying to solve a few last riddles that Sphinx is asking me to solve, > getting PROD image in order and having to make sure that several > distributions produced are equivalent to the single distribution we had so > far when installed together. That all seems doable today. Most of the > tests, scripts, release process and tooling updates, development workflows, > breeze image building and tests are already fixed in the PR. I will also > add more detailed description what's "in" the PR - I had to make a number > of small changes on how we are importing stuff and where dependencies are > kept etc (hint - it's going to be much, much more standard) but those are > mostly impacting development environment, rather than released > distributions. > > I will have a few follow-up things after that - but they will be far > smaller and far less - potentially - disruptive. > > I keep on rebasing the change and while it's not **too** difficult, it > might be difficult for some people to rebase and resolve their conflicts, > also I expect a few teething issues - mostly around CI/ breeze, so I think > the following plan of making it happen is going to be the best: > > 1) I will make the PR green and hopefully reviewed by the end of tomorrow > - or generally by the time last beta is released (it's mostly renames and > fixing stuff to be used from new place - so even if it is a lot of > changes/files, it should be somewhat easy to review) > 2) I will merge it **right after** the beta is released. This means that I > will merge and resolve conflicts for all PRs merged before beta3 (so hurry > up! :) ) > 3) After that - everyone will be asked to do four things: > a) rebase their opened PRs to latest main (and solve conflicts) > b) run `*breeze ci image build --python 3.9*" (for doc building and > some static checks) > c) run `*uv self upgrade*` followed by `*uv sync --reinstall*` > (possibly with `*--all-packages*` if they work on both - airflow core and > providers at the same time. > d) update their IDE settings - if they use IDEs - to make sure > airflow-core/src and airflow/core/tests are available for code completion > and import resolution (there will be updated screenshots on how to do it) > > 4) With that -> everything ***SHOULD*** work without big problems but I > will make myself available to solve any teething issues (and I am sure > other team members of the "CI/DEV task force" will help as well). > > Also I have been somewhat out (speaking and taking part in a great L8 Conf > event in Warsaw Mon/Tue). I will be able to spend a lot of time Friday and > weekend to fix any potential teething issues - so that starting Monday next > week we can all continue working towards RC1 in an uninterrupted way. > > I am also going to be very available to fix any potential issues next week > - while I will be completing some left-over tasks (I hope to get Sphinx a > little less mysterious by moving the docs to a proper "devel-common" place > for one and start identifying DB tests in providers and paving a path on > how we can get rid of those). > > Let me know if you have any questions - I will do more docs and > explanation and also happy to talk on slack and explain more of the changes > once the PR is undrafted. > > J. > > > > On Fri, Mar 14, 2025 at 8:27 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> Hey here, >> >> I have a first (very draft and still requires a number of changes) PR for >> the final step of big refactoring of our projects and using workspace. This >> is to let you know about the changes coming (so please take a look at the >> consequences to not be surprised). >> >> This is the most *scary* one -> moving all airflow code to >> "airflow-core". And I have draft version of it in >> https://github.com/apache/airflow/pull/47798 >> >> And it's not for the faint of heart :) >> >> [image: image.png] >> >> Note! It's not yet complete and unless you have some general comments, >> it's likely not worth pointing to individual changes (yet) - it's more to >> take a look at how things will look like eventually. I will work in the >> next two days to get it to reviewable state, and will keep it rebased and >> running till mid next-week. I would like to have it ready (including the >> release process) for the fourth (and final?) beta). >> >> Some resulting packaging changes: >> >> *FOR DEVELOPMENT:* >> >> * the pyproject.toml in the "root" of Airflow is still "apache-airflow" >> package - but this will be an empty "meta" package that will install >> together "apache-airflow-core", "apache-airflow-task-sdk" and optionally >> providers (via extras) >> >> * the airflow-core is a new "apache-airflow-core" distribution, where >> only airflow dependencies and airflow "core" extras are configured (smtp/ >> otel, pandas,rabbitmq etc) - I will likely cleanup some of those as well, >> some of them are not needed. the nice thing is that this package has all >> dependencies static (no hatch_build.py - everything is in pyproject.toml) - >> which is pretty cool and allow us to better use dependabot for security >> upgrades and notifications >> >> The airflow-core structure is pretty standard: >> >> airflow-core # <- this is folder where airflow-core distribution is >> \- src >> | \ airflow # <- This is airflow package >> | \- api >> | |- api_fastapi >> | |- assets >> | ... >> |- tests >> | \- always >> | |- api >> | ... >> |- docs >> | >> |- pyproject.toml >> |- README.md >> >> >> * for development - i will describe later the `pypi` way, but with `uv` >> things get simpler and we have a few new options (Dennis - this is >> continuation of discussion on the uv sync commands, so it's worth to >> look closely: >> >> There are a number of ways you will be (eventually able to interact with >> venv. After you checkout Airflow. You can change working directory and work >> on different packages and depending on which directory you run `uv sync` - >> uv (using workspace feature) will sync the **expected** dependencies. >> >> It's best to get used to the fact that instead of one airflow project we >> will have ~100 pretty independent projects, and while you can continue >> working with all of them as a single huge "workspace", it is generally way >> more convenient to change directory to the "distribution" you are working >> on currently and do everything there - with isolated set of dependencies >> required only for that "distribution" - "airflow-core", "task-sdk", >> "providers/amazon", "providers/mongo" - those are all separate >> distributions, and more and more we will be able to treat them as >> independent projects (but we will conveniently keep the option to develop >> and run tests in a joined "workspace" environment at the top of the project >> where we can install and test everything together - that's a bit of `uv >> workspace` magic in play. >> >> Here are typical patterns: >> >> 1) Installing all development dependencies for everything (I.e complete >> environment like in breeze) -- allows to run all tests for all airflow and >> all providers >> >> cd . >> uv sync --all-packages >> >> 2) installing just airflow core with required dependencies (ready for >> most core tests) >> >> cd airflow-core >> uv sync >> >> 3) installing airflow core with optional dependencies (should allow to >> run all core tests - including for the optional core features such as otel >> etc). >> >> cd airflow-core >> uv sync --all-extras >> >> 4) installing individual provider dependencies (say amazon) - this allows >> to run all tests of the provider you are working on - including installing >> all dependencies from cross-provider dependencies (i.e. if you have google >> tests in amazon provider, it will also install necessary google >> dependencies). >> >> cd providers/amazon >> uv sync >> >> Generally speaking - "airflow-core" will become (eventually) a truly >> airflow-only distribution. It will have a few dependencies to "standard" >> and "fab" providers - but I hope we will be able to get rid of those during >> the resulting cleanup. >> >> The IDE (IntelliJ) setting will just require "airflow-core/src" and >> "airflow-core/tests" to be source/test roots as usual for other >> distributions. >> >> I will update the docs after I complete the PR, there are some small >> variations on when to install which extras and I will play a bit to get to >> the best developer experience and least surprises. >> >> *FOR USERS* >> >> For "installable" airflow (i.e. user's experience) - the changes will be >> pretty much 100% transparent. When user will install "apache-airflow" or >> "apache-airflow[google]" - things will work as they did before - only >> instead of one "apache-airflow" distribution, they will have >> "apache-airflow", "apache-airflow-core" and "apache-airflow-task-sdk" >> installed. >> >> Regarding version numbers etc., I will start a separate discussion - >> later next week after we see how those packages will interact >> ("apache-airflow" will only contain extras, but for compatibility reasons >> we likely want to pin both "apache-airflow" and "apache-airflow-core" to >> each other, so that users will be able to upgrade "core" by upgrading >> "apache-airflow" - we do not want to change those habits likely. >> >> The "apache-airflow-task-sdk" will be versioned separately. >> >> Please take a look - also at the PR, see if you have any big >> issues/questions/doubts - let's start discussion here - I am happy to >> answer all general questions and adapt the PR to respond to >> questions/suggestions. >> >> In the meantime I will be working on making the PR green and adding >> missing bits and pieces for the release process. >> >> J. >> >> >> >>
