Thank you so much for this Jarek, incredible effort. uv really has been a lifesaver and I can't wait to now try out the simplified dev workflow.
Regards, Kunal Bhattacharya On Thu, Apr 3, 2025 at 11:16 AM Amogh Desai <amoghdesai....@gmail.com> wrote: > Thank you for the effort Jarek! > > I am looking forward to having a simpler life building docs. Personally it > was a nightmare for me earlier. > Will shout out if / when I get to that stage. > > Thanks & Regards, > Amogh Desai > > > On Wed, Apr 2, 2025 at 10:37 PM Buğra Öztürk <ozturkbugr...@gmail.com> > wrote: > > > Amazing news! Thanks for the huge effort Jarek! :) > > > > On Wed, 2 Apr 2025, 18:45 Pavankumar Gopidesu, <gopidesupa...@gmail.com> > > wrote: > > > > > That's awesome Jarek, thank you for this :) > > > > > > Regards, > > > Pavan > > > > > > On Wed, Apr 2, 2025 at 1:09 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > > You might also want to do *docker system prune* or even *docker > system > > > > prune --all* or run *breeze doctor* to clean-up some stale cache, > > > images, > > > > docker volumes > > > > > > > > Also IntelliJ/PyCharm users *uv run setup_idea.py* will add some > > missing > > > > directories and regenerate your IntelliJ project configuration. > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Apr 2, 2025 at 1:11 PM Jarek Potiuk <ja...@potiuk.com> > wrote: > > > > > > > > > Hello here, > > > > > > > > > > As part of the packaging work - I merged the > > > > > https://github.com/apache/airflow/pull/48223 > > > > > > > > > > *TL;DR; Rebase all PRS, rebuild images and uv sync - and > > > > > hopefully everything should work as before even if a lot of things > > > moved. > > > > > Hopefully the move will be largely transparent (except changing > > > *include: > > > > > in open PRs in docs). * > > > > > > > > > > *NOTE! UV is now mandatory and a lot of code is gone thanks to > that. > > > > > Breeze will also refuse to work if uv is not installed.* > > > > > > > > > > It took a bit of time, but we are in a much more standard and > better > > > > shape > > > > > now - and as a side effect (which was intended but I had to > implement > > > it > > > > as > > > > > part of that monster PR to fix documentation) we now have a much > > > simpler > > > > > (more guidelines are coming) way to iterate on doc building. > > > > > > > > > > *Few important things first: * > > > > > > > > > > 1) Make sure to rebase your PRs, Run `breeze image build`, Rnu `uv > > > sync`. > > > > > Due to the way git handles things - you ** might ** have some > > dangling > > > > > generated directories in your repo and they might cause some > > problems. > > > > Run > > > > > "git status" after rebase and see you have some files you need to > > > delete > > > > > (manually) > > > > > > > > > > 2) If you are brave enough - you might want to run `breeze doctor` > > and > > > > > cleanup git repo - it should clean all files that should be > removed, > > > but > > > > > also it might remove some of your custom configurations and files > you > > > > > created, > > > > > > > > > > 3) Generally everything should work as it worked before with breeze > > > (for > > > > > example `breeze build-docs` command works as before. But a number > of > > > > > folders/distributions/code (not airflow nore providers directly) > were > > > > > moved/updated. For now you can just continue to build docs as > before > > - > > > > with > > > > > breeze. But simpler/faster ways are coming as follow up. > > > > > > > > > > 4) If you have some new examples or documentation included in your > > PRs > > > > the > > > > > doc build might start falling for you - but this is because > > `include::` > > > > or > > > > > `exampleinclude:` might need to be updated - look at other > examples > > - > > > I > > > > > fixed the includes in all providers. More explanation in follow-up > > doc > > > > > build improvement PR - in the meantime, feel free to ask on slack > or > > PR > > > > for > > > > > help. > > > > > > > > > > *Generated provider_dependencies.json do not need to be updated* > > > > > > > > > > The "generated/provider_dependencies.json" is no longer committed > to > > > the > > > > > repo - it is .gitignored. We are generating it as-needed on the > > flight. > > > > It > > > > > should be automatically regenerated when you run pre-commits > locally > > > and > > > > > when you build the breeze image. > > > > > > > > > > There might be some cases when we add dependencies and you will > need > > to > > > > > regenerate it but that should happen automatically as needed. > > > > > > > > > > *New, updated folders* > > > > > > > > > > The change are mostly with these: > > > > > > > > > > ./dev/pyproject.toml > > > > > ./devel-common/pyproject.toml > > > > > ./doc > > > > > ./docker-stack-docs > > > > > ./providers-summary-docs > > > > > > > > > > *More explanation for distributions/folder changes* > > > > > > > > > > The dev is now a separate distribution with its own pyproject.toml > > > > > dependencies that are used for all the release management and > general > > > dev > > > > > housekeeping. This is different from "devel-common" which is a > common > > > > > package with a lot of common code reused for tests and builds > > > (including > > > > > doc builds scripts are using). > > > > > > > > > > We might want to change names later, as they are confusing but I > will > > > > > leave that discussion for later when we complete all the isolation > > > work - > > > > > with those changes I implemented it will be very easy (as opposed > to > > > how > > > > it > > > > > was) to move those distributions around - and if we will want to > > > > > restructure it again, that will be a very simple move. > > > > > > > > > > *Improved doc structure* > > > > > > > > > > This is because we finally made the last step - where the "doc" > code > > is > > > > > moved to "devel-common" and we are importing it from there, also > each > > > > > distribution has it's own "conf.py" and that makes modifying docs > > > > building > > > > > and our sphinx building scripts MUCH more readable ana mangeable. > The > > > > "doc" > > > > > code now contains just spelling wordlist and README.md explaining > > where > > > > to > > > > > look for the documentation. > > > > > > > > > > Those new top-level folders ("docker-stack-docs" and > > > > > "providers-summary-docs") - contain the two independent pieces of > our > > > > > documentation ("docker-stack" and "apache-airflow-providers" > package > > > > names > > > > > from "breeze build-docs" command. > > > > > > > > > > The REALLY nice thing now is that the doc files are not copied > > between > > > > > places and each of the docs folder (including each of the 90 > > providers > > > > docs > > > > > folder) has it's own "conf.py", which makes the "docs" folder for > > them > > > > > "isolated" and "fully self-contained" - which will open up the ways > > on > > > > much > > > > > more friendly sphinx workflows. Wait for it ! Coming **just in > time** > > > > when > > > > > we most need it - when we want to start updating the docs for > Airflow > > > 3. > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
