Hello here,
As part of the packaging work - I merged the
https://github.com/apache/airflow/pull/48223
*TL;DR; Rebase all PRS, rebuild images and uv sync - and
hopefully everything should work as before even if a lot of things moved.
Hopefully the move will be largely transparent (except changing *include:
in open PRs in docs). *
*NOTE! UV is now mandatory and a lot of code is gone thanks to that. Breeze
will also refuse to work if uv is not installed.*
It took a bit of time, but we are in a much more standard and better shape
now - and as a side effect (which was intended but I had to implement it as
part of that monster PR to fix documentation) we now have a much simpler
(more guidelines are coming) way to iterate on doc building.
*Few important things first: *
1) Make sure to rebase your PRs, Run `breeze image build`, Rnu `uv sync`.
Due to the way git handles things - you ** might ** have some dangling
generated directories in your repo and they might cause some problems. Run
"git status" after rebase and see you have some files you need to delete
(manually)
2) If you are brave enough - you might want to run `breeze doctor` and
cleanup git repo - it should clean all files that should be removed, but
also it might remove some of your custom configurations and files you
created,
3) Generally everything should work as it worked before with breeze (for
example `breeze build-docs` command works as before. But a number of
folders/distributions/code (not airflow nore providers directly) were
moved/updated. For now you can just continue to build docs as before - with
breeze. But simpler/faster ways are coming as follow up.
4) If you have some new examples or documentation included in your PRs the
doc build might start falling for you - but this is because `include::` or
`exampleinclude:` might need to be updated - look at other examples - I
fixed the includes in all providers. More explanation in follow-up doc
build improvement PR - in the meantime, feel free to ask on slack or PR for
help.
*Generated provider_dependencies.json do not need to be updated*
The "generated/provider_dependencies.json" is no longer committed to the
repo - it is .gitignored. We are generating it as-needed on the flight. It
should be automatically regenerated when you run pre-commits locally and
when you build the breeze image.
There might be some cases when we add dependencies and you will need to
regenerate it but that should happen automatically as needed.
*New, updated folders*
The change are mostly with these:
./dev/pyproject.toml
./devel-common/pyproject.toml
./doc
./docker-stack-docs
./providers-summary-docs
*More explanation for distributions/folder changes*
The dev is now a separate distribution with its own pyproject.toml
dependencies that are used for all the release management and general dev
housekeeping. This is different from "devel-common" which is a common
package with a lot of common code reused for tests and builds (including
doc builds scripts are using).
We might want to change names later, as they are confusing but I will leave
that discussion for later when we complete all the isolation work - with
those changes I implemented it will be very easy (as opposed to how it was)
to move those distributions around - and if we will want to restructure it
again, that will be a very simple move.
*Improved doc structure*
This is because we finally made the last step - where the "doc" code is
moved to "devel-common" and we are importing it from there, also each
distribution has it's own "conf.py" and that makes modifying docs building
and our sphinx building scripts MUCH more readable ana mangeable. The "doc"
code now contains just spelling wordlist and README.md explaining where to
look for the documentation.
Those new top-level folders ("docker-stack-docs" and
"providers-summary-docs") - contain the two independent pieces of our
documentation ("docker-stack" and "apache-airflow-providers" package names
from "breeze build-docs" command.
The REALLY nice thing now is that the doc files are not copied between
places and each of the docs folder (including each of the 90 providers docs
folder) has it's own "conf.py", which makes the "docs" folder for them
"isolated" and "fully self-contained" - which will open up the ways on much
more friendly sphinx workflows. Wait for it ! Coming **just in time** when
we most need it - when we want to start updating the docs for Airflow 3.
J.
