Fantastic work everyone! Pavan
On Wed, Feb 12, 2025 at 10:40 PM Vikram Koka <vik...@astronomer.io.invalid> wrote: > Amazing work team! > Kudos to everyone who contributed to this effort. > > It was wonderful to see this coming together and I was amazed how quickly > all of you made this happen. > > Vikram > > > On Tue, Feb 11, 2025 at 9:10 AM Shahar Epstein <sha...@apache.org> wrote: > > > Thanks Jarek and everyone involved, great step! > > > > > > Shahar > > > > On Sun, Feb 9, 2025, 11:40 Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > Hello, > > > > > > I have a pleasure to announce that the provider's move to the new > > structure > > > is now complete. We have no more providers in "providers/src". Many, > many > > > thanks to all those who helped, truly collaborated and learned from > each > > > other during the migration process. This was quite a journey, where I > > > set-off the migration, and went to Brussels for almost a week of > > conference > > > where I had very little time, yet things were moving with the lightning > > > speed with only a little of my help and encouragement. > > > > > > The Airflow community is the best! > > > > > > Thanks to - In no particular order: > > > > > > > > > *Kalyan, Pratiksha, Elad, Rahul, Shubham, Kunal, Josix, Bugra, LIU ZHE > > YOU, > > > Amogh, Aritra, Nikolas, got686-yandex, David Blain, Ambika, Idris, > Ankit, > > > Mikhail Dengin, Dennis, Jens * > > > And anyone else who I missed. This has been fantastic teamwork :). So > > many > > > people got involved and helped. > > > > > > *THANK YOU! * > > > > > > *What do we have now?* > > > > > > Each provider now has its own pyproject.toml file and is effectively a > > > separate sub-project in our monorepo. There are few things it enables a > > few > > > things: > > > > > > a) you can easily build each provider now with just `hatch build .` or > > > `flit build .` or any other frontend - making the providers "modern > > > standard PEP-compliant" > > > b) you can install the "main" (or any other branch or commit) version > of > > > the provider using github URL. This for example allows for easy testing > > of > > > not-yet-released providers: any of the "developer-focused" users who > > would > > > like to use the "main" version with changes they introduced for > example, > > > could install such pre-release providers in their environment very > > easily > > > now. > > > c) we can now start proceeding with next steps - making core truly > > > independent from providers (there are still some references, tests and > > > dependencies left) and proceed with further simplifying of our CI and > > > turning all db-tests in providers into non-db tests (to make sure that > > they > > > are not dependent on the DB while we switch to Task SDK) - following > > steps > > > 2-4 outlined in > > > https://github.com/apache/airflow/issues/42632#issuecomment-2449671014 > > > d) removal of a lot of code that handled the old ways of doing things > > where > > > sources of providers were shared with Airflow. > > > > > > > > > *One watchout !!!!: * > > > Currently on MacOS you can hit `*too many open files*` errors when > > running > > > `uv sync`. This issue is being worked on by Astral team in > > > https://github.com/astral-sh/uv/issues/11296 (they are happy to have > > > airflow again stretching the limits of `uv` - as they wrote "airflow is > > > their favourite benchmark and test case"). This is in essence caused > by a > > > very low limit set by default on the number of opened files by MacOS > > (256). > > > > > > It is easily mitigated by adding `*ulimit -n 2048*` in your .bashrc or > > > .zshrc and we described it in the docs. but it would be nice to have it > > > fixed in `uv` eventually and get `uv sync` works out-of-the-box for > > Airflow > > > - I am quite sure that the Astral team will fix it soon. For now I > added > > an > > > explanation in > > > > > > > > > https://github.com/apache/airflow/blob/main/contributing-docs/07_local_virtualenv.rst > > > and will further clarify that it should be done in your .rc file to be > > > persistent. > > > > > > > > > *What's coming?* > > > What's next is a cleanup. We still have quite a lot of duplicated code > to > > > remove, and few places where we still manually emulate `uv workspace` > > > rather than use it. > > > > > > > > > *Personal note* > > > It's been quite a journey for me personally. > > > > > > Ash had always "complained" about the current setup and we both agreed > > that > > > having a "proper" monorepo with separate sub-projects is a good thing > to > > > have. But the tooling was not there. The standards were not there for > > > years. Python packaging PEPs implemented in the last few years and > > tooling > > > improvements (notably `uv workspace` that I helped Charlie and Astral > > team > > > to design to fit our case) had to catch-up, and the last few years > Python > > > packaging had improved immensely and it's picking up speed. I made my > > first > > > POC to move the providers in December 2022: > > > https://github.com/apache/airflow/pull/28292 and the first email on > the > > > devlist I sent about it was 12th December 2022: > > > https://lists.apache.org/thread/3s5tn1wnvo0cw9vofwmbjl0rkyvhrtbx . But > > > back > > > then it would be far too complex for our contributors to use, without > all > > > the tooling support and standards. > > > > > > TP particularly, who is a packaging team committee has been driving a > lot > > > of those in the Packaging team and he deserves an absolute shout-out > > here. > > > He is a bit of a silent hero who discusses and participates in many > PEPs > > > that we make use of. > > > > > > But even though it was me who mostly pushed and pulled many strings > > around > > > it - and TP who was actively participating in the process - it was all > > > community effort. We not only patiently waited for it but also actively > > > helped to move the standards, encouraged them and helped others to > > > implement features that we needed. So it's more than 2 years of intense > > > work of packaging team, introduction of new tool (`uv`) in packaging > > space > > > and us making incremental improvements, switching to modern PEP > standards > > > in December 2023 and many other small things that could be seen as > "yacc > > > beating" as some might call it, but eventually were needed those many > > > smaller and bigger things to get here. > > > > > > *And the journey is absolutely not over:* > > > > > > I am also looking forward to what's coming and I am also planning to > help > > > in Python community and get involved (and help to shape) a few other > > things > > > that are in progress that will (finally) catch-up with what Airflow > needs > > > are, so that we can finally get rid of even more custom code we have > and > > > improve both development and security of our processes and reflect more > > the > > > way we (and the Apache Software Foundation works), I hope to have some > > more > > > time after we complete the current packaging work to help with those - > i > > > promised it in a few of those, but I had to yet deliver my promise. And > > > also anyone in the community here is welcome to help as well, as you > see, > > > it eventually pays off. > > > > > > * https://peps.python.org/pep-0751/ -> *A file format to record Python > > > dependencies for installation reproducibility *-> this will finally > > codify > > > what we do as a "poor man's" solution with constraints. I've been > waiting > > > for that one to be there for years, and there was a rejected version of > > it > > > (TP participated in it) - but it looks like we are getting there to > make > > it > > > a "standard" that we - and tooling out there - will just be able to > > follow > > > * https://peps.python.org/pep-0752/ -> *Implicit namespaces for > package > > > repositories* -> will be helpful for naming of our packages in PyPI to > be > > > consistent and not hi-jacked > > > * https://peps.python.org/pep-0770/ *-> Improving measurability of > > Python > > > packages with Software Bill-of-Materials* - where we will be able to > > embed > > > our SBOMS we already generate in PyPI metadata > > > * https://peps.python.org/pep-0771/ -> *Default Extras for Python > > Software > > > Packages* - which will allow us to get rid of our custom "preinstalled > > > packages" > > > * https://peps.python.org/pep-0735/ -> *Dependency Groups in > > > pyproject.toml* > > > - which we already partially use, but once `pip` releases it (already > > > merged and planned to be released in 25.1 - will allow us to replace > our > > > `extras` with dependency groups for development > > > > > > ... and more to come .... > > > > > > All these things we need for our workflows and setup and so far we had > to > > > do some "custom" band-aid solutions, but the awesome packaging team is > > > discussing and implementing things to make all those "first class > > citizens" > > > in Python packaging and it will let us switch to those. > > > > > > Looking forward to all those improvements in the (near) future. Looks > > like > > > the next few years will keep me (and others) busy with those. > > > > > > J. > > > > > >
