After the initial move of few providers - the move is in progress https://github.com/apache/airflow/issues/46045 , also we have a dedicated channel in our Slack: https://apache-airflow.slack.com/archives/C08AA9EJFB5. -> #new-providers-strucuture Already few people are moving providers (Thanks Kalyan, Rahul and Pratiksha !).- but anyone is free to join the efforts. The instructions on what to do are in https://github.com/apache/airflow/blob/main/dev/moving_providers/README.md and linked from the issue above.
I also encourage everyone who has troubles with getting their environment up and running with providers to also reach out to the #new-providers-structure channel, there we will be helping each other to solve any teething issues. We are trying to see if there are some outstanding PRs for respective providers, and try to refrain from merging them before the authors will have a chance to rebase and fix them - and I am also going to some two weeks worth PRs and will either merge things where ready or let the authors know that they should speed up if they want to avoid conflicts. I hope it would not take longer than few days/week to complete the process - I am in Brussels for FOSDEM and "surrounding events" the whole next week starting Tuesday night, but will keep a watchful eye on the progress and do some of it myself and help in case of issues that will need manual resolution. There will be a number of small cases where the automation is doing 95% of migration and some small things have to be fixed/adjusted easily manually (or so I hope) - that's why I have not done it as a single big PR - because it's easier to get people work on them in parallel separately in each provider than try to solve all of them in one big PR. >From the user's point of view: except the location of the provider's folder (instead of being part of "providers/src" each provider has it's own folder directly under "providers" and consolidating all "provider things" like "tests" and "docs" in one "per-provider" folder, the only change is on how you create your own local venv. * HIGHLY RECOMMENDED: If you already switched to `uv` and using `.venv` environment created by it automatically - nothing changes for you: `uv sync` will sync basic airflow dependencies to the .venv, `uv sync --alle-extras` will sync all dependencies for airflow + providers, `uv sync --extra celery --extra google` etc. will sync the appropriate extras. Note - that when we complete, you will not even need to use extras, I think. all dependencies should be created and sync by default (and you can exclude them with other flags `--no-install-wrkspace` for one. Details of some of the workflows there will need some follow-ups later after we complete the migration, and I will add some more docs on it later - it will also change later at the next step when we split packages for Airflow 3 a bit more and a bit differently than today - then the `workspace` feature will become even more useful and some of the development /environment setup tasks will be **easy** (well - there will be some teething problems, I am sure - but it looks very promising to make our development workflows simpler and more straightforward so far). Note that you will need to do `uv sync` manually after we migrate new providers - so likely your `uv sync` will have to be run every time you rebase to the latest main over then next week or so. (but it's super quick). * If you still want to use pip - you are covered. The only thing you will need to do is to manually install required packages in --editable mode: `pip install -e . -e ./task-sdk -e ./providers/. -e ./providers/celery -e ./providers/google` etc. - while `pip` does not make use of the workspace feature we have with `uv` - the `workspace` feature is based on `PEP` packaging standards, so it's just a little bit more complex to get your packages installed together. This is all already described in https://github.com/apache/airflow/blob/main/contributing-docs/07_local_virtualenv.rst The Breeze workflow (except location of the files) is unaffected. Things should **just work** - as long as you respond "y" when asked whether to rebuild the image or when you run `breeze ci-image build` manually after rebasing (and continue doing it while we are merging the PRs). So similarly as `uv sync` - rebuilding the images might need to be done more often (but also that one uses now `uv` and local caching, so it should be relatively quick - no longer than 20-30 seconds usually for me. Looking forward to completing the move quick and relatively painlessly J. On Mon, Jan 20, 2025 at 10:22 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Ah and one comment (Pierre already encountered it) - before running > `breeze start-airflow` you will need to rebuild the image. > Breeze will nicely ask you to do so, so pressing `y` when it does will do > the job. > > This will happen more as we move the providers, so for a while if you see > missing imports or similar - rebuild the image first. > > On Mon, Jan 20, 2025 at 6:32 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> Update: >> >> We have now 3 providers moved: *airbyte, celery and edge - pre-release* >> (thanks Jens for moving edge one!). I have already fixed a few initial >> issues and improvements and added some consistency checks, there are two or >> three more providers to move with some of the things I still need to add >> (like devel deps) and we need to do a release cycle with providers to see >> if everything works in the new structure, once we do it, we can accelerate >> migration of the rest - I will prepare some scripts that will help with it. >> >> For now - one thing to remember: if you are adding/modifying dependencies >> in the moved providers, they are no longer updated in *provider.yaml*, >> you should edit them directly in respective `*pyproject.toml*`, For >> example for celery, it will be `providers/celery/pyproject.toml` -> that >> should work for required, optional and soon provider's devel dependencies >> as well. Even if that file is automatically generated and some dependencies >> (cross-provider ones) are added automatically if missing, any changes to >> dependencies are preserved (including comments). I added some information >> about it in our contributing docs: >> https://github.com/apache/airflow/blob/main/contributing-docs/11_provider_packages.rst#where-providers-are-kept-in-our-repository >> >> Also apologies in advance to those who will have PRs in progress in >> providers, once we move the provider, you will have to rebase your changes >> and use the new location of provider src/ test/ doc/ dirs - this should be >> rather painless, but I know it's annoying. That's why we also will migrate >> provider-by-provider and spread it a bit to avoid big-bang and huge >> disruption for everyone. >> >> J. >> >> >> >> On Tue, Jan 14, 2025 at 4:50 PM Jarek Potiuk <ja...@potiuk.com> wrote: >> >>> I will try to get celery next - we need a bugfix for it after yanking >>> the last release. >>> >>> On Tue, Jan 14, 2025 at 9:46 AM Elad Kalif <elad...@apache.org> wrote: >>> >>>> I plan to cut a wave soon but we can't test on airbyte as there were no >>>> changes since last release >>>> >>>> On Tue, Jan 14, 2025 at 8:52 AM Jarek Potiuk <ja...@potiuk.com> wrote: >>>> >>>> > First provider (airbyte) merged. Managed to walk around Sphinx without >>>> > answering too many riddles finally. >>>> > >>>> > In the next coming days I will migrate a few more providers and once >>>> we get >>>> > through a next provider release cycle with those few providers and >>>> fix/see >>>> > any kind of teething issues I will add semi-automation to move the >>>> provider >>>> > one-by one and I will ask for help in individually migrating all of >>>> them. >>>> > >>>> > The two limitations that will need to be solved later (so the provider >>>> > projects are not yet truly standalone) >>>> > >>>> > * tests are only running from within airflow - you cannot run tests >>>> inside >>>> > `providers/airbyte" yet >>>> > * doc build also currently works only from "airflow" main repo. - you >>>> can't >>>> > build docs as "command" in the build of the provider project >>>> > >>>> > Both are solvable but require some more reshuffling and sphinx scripts >>>> > rewriting basically, so something that will be way easier when we >>>> migrate >>>> > all providers and remove the old provider approach. >>>> > >>>> > I will summarize the changes you should expect when contributing to a >>>> > provider in a separate mail once I complete a few more providers. >>>> > >>>> > J. >>>> > >>>> > On Fri, Jan 10, 2025 at 5:52 PM Vincent Beck <vincb...@apache.org> >>>> wrote: >>>> > >>>> > > Same, I am not strongly opinionated, just a preference :) >>>> > > >>>> > > On 2025/01/10 15:36:05 Vikram Koka wrote: >>>> > > > I agree this makes sense. >>>> > > > >>>> > > > I was originally concerned that this would make it more difficult >>>> to >>>> > > ensure >>>> > > > compatibility across providers for capabilities such as >>>> common.sql, >>>> > > > objectstore, and so on. >>>> > > > However, seeing that the "common" pattern would remain the same >>>> and >>>> > it's >>>> > > > only the code layout that is changing, and that we are getting >>>> rid of a >>>> > > ton >>>> > > > of generated code, I am positive on this. >>>> > > > >>>> > > > >>>> > > > On Fri, Jan 10, 2025 at 3:07 AM Jarek Potiuk <ja...@potiuk.com> >>>> wrote: >>>> > > > >>>> > > > > So I propose letting the "doer" make the decision if we are >>>> split. >>>> > > > > >>>> > > > > On Fri, Jan 10, 2025 at 11:52 AM Ash Berlin-Taylor < >>>> a...@apache.org> >>>> > > wrote: >>>> > > > > >>>> > > > > > Not strong at all, preference Is all. It sounds like Vincent >>>> and I >>>> > > are in >>>> > > > > > the hyphen camp and you and Maciej are in the slash camp. >>>> > > > > > >>>> > > > > > +1 on the “I don’t care what code style is used as long as it >>>> is >>>> > > > > > programmatically enforced”. >>>> > > > > > >>>> > > > > > -a >>>> > > > > > >>>> > > > > > > On 10 Jan 2025, at 09:41, Jarek Potiuk <ja...@potiuk.com> >>>> wrote: >>>> > > > > > > >>>> > > > > > > Is there anything else that "tastes" Ash ? A concrete >>>> reason that >>>> > > makes >>>> > > > > > you >>>> > > > > > > think the "-" prefix in this case is better than the "/" >>>> folder? >>>> > > How >>>> > > > > > > strong is your "taste" preference and do you think it will >>>> have >>>> > > some >>>> > > > > > > lasting effect if we choose to flatten the folder structure? >>>> > > > > > > >>>> > > > > > > I might make a small vote to see what is the preference of >>>> people >>>> > > if we >>>> > > > > > > think this is an important aspect. >>>> > > > > > > >>>> > > > > > > BTW. This is why I really love black/ruff formatting - we >>>> stopped >>>> > > > > wasting >>>> > > > > > > time on "taste" discussion - it does not matter what is the >>>> > > individual >>>> > > > > > > preference, consistency is more important and I prefer to do >>>> > stuff >>>> > > that >>>> > > > > > > really matters but if people feel strongly that we should >>>> discuss >>>> > > it, I >>>> > > > > > > might make a vote there. >>>> > > > > > > >>>> > > > > > > J. >>>> > > > > > > >>>> > > > > > > On Thu, Jan 9, 2025 at 5:48 PM Ash Berlin-Taylor < >>>> a...@apache.org >>>> > > >>>> > > > > wrote: >>>> > > > > > > >>>> > > > > > >> My preference is for being “more direct” and not having >>>> deeply >>>> > > nested >>>> > > > > > >> things where possible — I think Microsoft might be the one >>>> case >>>> > > where >>>> > > > > > >> having extra folders makes sense. And I’m fine with things >>>> not >>>> > > being >>>> > > > > > >> consistent across providers/groups of providers. >>>> > > > > > >> >>>> > > > > > >> -ash >>>> > > > > > >> >>>> > > > > > >> >>>> > > > > > >> >>>> > > > > > >>> On 8 Jan 2025, at 17:18, Jarek Potiuk <ja...@potiuk.com> >>>> > wrote: >>>> > > > > > >>> >>>> > > > > > >>> Can you give an example of what might break why having >>>> > > > > > >>> `providers/aapche-beam/src/airflow/providers/apache/beam`? >>>> > > > > > >>> >>>> > > > > > >>> Nothing will break. It's just: >>>> > > > > > >>> >>>> > > > > > >>> * the code will have to be a little more complex as it >>>> will >>>> > have >>>> > > to >>>> > > > > do >>>> > > > > > >> some >>>> > > > > > >>> conditional writes of "-" "/" >>>> > > > > > >>> * there will be inconsistency in the depth of folders - >>>> outside >>>> > > it >>>> > > > > will >>>> > > > > > >> be >>>> > > > > > >>> 1, inside it will be 2 (as it is in your example)/ >>>> > > > > > >>> * it will be a bit more convention/ complex to limit >>>> related >>>> > > > > providers >>>> > > > > > >> (say >>>> > > > > > >>> microsoft) - with the current scheme >>>> "providers/microsoft" is >>>> > the >>>> > > > > > >> directory >>>> > > > > > >>> containing all microsoft providers. If we change it to >>>> "-", you >>>> > > have >>>> > > > > to >>>> > > > > > >>> find all sub-directories following "microsoft-*" >>>> convention. >>>> > > > > > >>> >>>> > > > > > >>> I am not super-strong on it - we could do either, it's >>>> just my >>>> > > > > > preference >>>> > > > > > >>> to use folders for grouping related things (as folders >>>> were >>>> > > designed >>>> > > > > > >> for). >>>> > > > > > >>> >>>> > > > > > >>> J. >>>> > > > > > >>> >>>> > > > > > >>> On Wed, Jan 8, 2025 at 5:03 PM Ash Berlin-Taylor < >>>> > a...@apache.org >>>> > > > >>>> > > > > > wrote: >>>> > > > > > >>> >>>> > > > > > >>>>> And we already have a number of mappings and >>>> conventions to >>>> > > handle >>>> > > > > > >> that. >>>> > > > > > >>>>> For example provider I'd mapping to dirs (apache.beam -> >>>> > > > > > apache/beam), >>>> > > > > > >>>> and >>>> > > > > > >>>>> 'apache-airflow-providers-apache-beam' as package na e >>>> and >>>> > > > > > >>>>> airflow/providers/apache/beam as packages inside the >>>> > > distribution. >>>> > > > > > >> Those >>>> > > > > > >>>>> will remain as they are - we cannot change them without >>>> > > breaking >>>> > > > > > >>>>> compatibility. >>>> > > > > > >>>> >>>> > > > > > >>>> Can you give an example of what might break why having >>>> > > > > > >>>> >>>> `providers/aapche-beam/src/airflow/providers/apache/beam`? >>>> > > > > > >>>> >>>> > > > > > >>>> -a >>>> > > > > > >>>> >>>> > > > > > >>>>> On 7 Jan 2025, at 18:33, Jarek Potiuk <ja...@potiuk.com >>>> > >>>> > > wrote: >>>> > > > > > >>>>> >>>> > > > > > >>>>> I think it will be better to keep it. >>>> > > > > > >>>>> >>>> > > > > > >>>>> The reason we have varying levels were to group things >>>> > > together - >>>> > > > > > >> mainly >>>> > > > > > >>>>> Apache related providers, but also Microsoft. >>>> > > > > > >>>>> >>>> > > > > > >>>>> And we already have a number of mappings and >>>> conventions to >>>> > > handle >>>> > > > > > >> that. >>>> > > > > > >>>>> For example provider I'd mapping to dirs (apache.beam -> >>>> > > > > > apache/beam), >>>> > > > > > >>>> and >>>> > > > > > >>>>> 'apache-airflow-providers-apache-beam' as package na e >>>> and >>>> > > > > > >>>>> airflow/providers/apache/beam as packages inside the >>>> > > distribution. >>>> > > > > > >> Those >>>> > > > > > >>>>> will remain as they are - we cannot change them without >>>> > > breaking >>>> > > > > > >>>>> compatibility. >>>> > > > > > >>>>> >>>> > > > > > >>>>> So if we change it to a flat structure we will have some >>>> > > > > > >> inconsistencies >>>> > > > > > >>>> - >>>> > > > > > >>>>> in some cases it will be single folder in others >>>> (packages) >>>> > > those >>>> > > > > > will >>>> > > > > > >> be >>>> > > > > > >>>>> two folders. >>>> > > > > > >>>>> >>>> > > > > > >>>>> I think it will be more harm than good if we get rid of >>>> the >>>> > > > > 'folder' >>>> > > > > > >>>>> structures - some of the code in breeze will have to >>>> treat >>>> > > those >>>> > > > > > >>>>> differently as well. Nothing extraordinary and very >>>> complex >>>> > but >>>> > > > > more >>>> > > > > > >>>>> complex-ish than it should be - already on top of >>>> handling >>>> > > > > > potentially >>>> > > > > > >>>>> nested folders >>>> > > > > > >>>>> >>>> > > > > > >>>>> So my preference would be to stay with apache/beam - >>>> it's >>>> > just >>>> > > more >>>> > > > > > >>>>> consistently handling the case where provider packages >>>> can be >>>> > > > > > one-level >>>> > > > > > >>>>> nested >>>> > > > > > >>>>> >>>> > > > > > >>>>> J >>>> > > > > > >>>>> >>>> > > > > > >>>>> >>>> > > > > > >>>>> wt., 7 sty 2025, 19:00 użytkownik Vincent Beck < >>>> > > > > vincb...@apache.org> >>>> > > > > > >>>>> napisał: >>>> > > > > > >>>>> >>>> > > > > > >>>>>> Good question. I always found it confusing to have some >>>> > > providers >>>> > > > > at >>>> > > > > > >>>>>> different level. Examples: >>>> > > > > > >>>>>> - "airbyte" in "providers" directory (I would qualify >>>> it as >>>> > > > > > "regular" >>>> > > > > > >>>>>> provider) >>>> > > > > > >>>>>> - "hive" in "providers/apache" >>>> > > > > > >>>>>> - "amazon" in "providers" but which contains only one >>>> sub >>>> > > > > directory >>>> > > > > > >>>> "aws" >>>> > > > > > >>>>>> >>>> > > > > > >>>>>> I would be in favor of using "-" instead of "/" so >>>> that all >>>> > > > > > providers >>>> > > > > > >>>> are >>>> > > > > > >>>>>> at the same level. >>>> > > > > > >>>>>> >>>> > > > > > >>>>>> >>>> > > > > > >>>>>> On 2025/01/07 16:38:10 Ash Berlin-Taylor wrote: >>>> > > > > > >>>>>>> +1 one to this on general terms, it will hopefully >>>> reduce a >>>> > > lot >>>> > > > > of >>>> > > > > > >> the >>>> > > > > > >>>>>> boilerplate we need. >>>> > > > > > >>>>>>> >>>> > > > > > >>>>>>> As for the amazon/aws example specifically that does >>>> bring >>>> > > up a >>>> > > > > > >>>>>> question, should we have `/` or `-`.. to give some >>>> examples: >>>> > > > > > >>>>>>> >>>> > > > > > >>>>>>> cncf kubernetes: ./providers/cncf/kubernetes or >>>> > > > > > >>>>>> ./providers/cncf-kubernetes >>>> > > > > > >>>>>>> Apache hive: ./providers/apache/hive or >>>> > > ./providers/apache-hive >>>> > > > > > >>>>>>> AWS: ./providers/amazon/aws or ./providers/amazon-aws >>>> > > > > > >>>>>>> >>>> > > > > > >>>>>>> There is no requirement from python etc on one form >>>> or the >>>> > > other >>>> > > > > > (as >>>> > > > > > >>>>>> it’s just a folder, not part of the module name), so >>>> it’s >>>> > what >>>> > > > > ever >>>> > > > > > >>>> makes >>>> > > > > > >>>>>> most sense to us. >>>> > > > > > >>>>>>> >>>> > > > > > >>>>>>> Jarek and Dennis (and others): what are your >>>> preferences on >>>> > > these >>>> > > > > > >>>> styles? >>>> > > > > > >>>>>>> >>>> > > > > > >>>>>>> -ash >>>> > > > > > >>>>>>> >>>> > > > > > >>>>>>>> On 6 Jan 2025, at 22:51, Jarek Potiuk < >>>> ja...@potiuk.com> >>>> > > wrote: >>>> > > > > > >>>>>>>> >>>> > > > > > >>>>>>>> Oh. . And one other benefit of it will be that we >>>> will be >>>> > > able >>>> > > > > to >>>> > > > > > >> get >>>> > > > > > >>>>>> rid >>>> > > > > > >>>>>>>> of about 40% of the "Providers Manager" code. >>>> Currently, >>>> > in >>>> > > > > > >> Providers >>>> > > > > > >>>>>>>> manager we have a lot of "ifs" that make it possible >>>> to >>>> > use >>>> > > > > > >> providers >>>> > > > > > >>>>>> in >>>> > > > > > >>>>>>>> breeze and local environment from the sources. In >>>> > > "production" >>>> > > > > > >>>>>> installation >>>> > > > > > >>>>>>>> we are using "get_provider_info" entry points to >>>> discover >>>> > > > > > providers >>>> > > > > > >>>>>> and >>>> > > > > > >>>>>>>> discover if provider is installed, but when you use >>>> > current >>>> > > > > > >> providers >>>> > > > > > >>>>>>>> installed in Breeze to inside "airflow", we rely on >>>> > > > > > `provider.yaml` >>>> > > > > > >> to >>>> > > > > > >>>>>> be >>>> > > > > > >>>>>>>> present in the "airflow.providers.PROVIDER_ID" path >>>> - so >>>> > we >>>> > > > > > >>>> effectively >>>> > > > > > >>>>>>>> have two paths of discovering information about the >>>> > > providers >>>> > > > > > >>>>>> installed. >>>> > > > > > >>>>>>>> >>>> > > > > > >>>>>>>> After all providers are migrated to the new >>>> structure, all >>>> > > > > > providers >>>> > > > > > >>>>>> are >>>> > > > > > >>>>>>>> separate "distributions" - and when you run `uv sync` >>>> > > (which >>>> > > > > will >>>> > > > > > >>>>>> install >>>> > > > > > >>>>>>>> all providers thanks to workspace feature) or `pip >>>> install >>>> > > -e >>>> > > > > > >>>>>>>> ./providers/aws` (which you will have to do manually >>>> to >>>> > > work on >>>> > > > > > the >>>> > > > > > >>>>>>>> provider - if you use `pip` rather than uv) - then >>>> we will >>>> > > not >>>> > > > > > have >>>> > > > > > >> to >>>> > > > > > >>>>>> use >>>> > > > > > >>>>>>>> the separate path to read provider.yaml, because the >>>> right >>>> > > > > > >> entrypoint >>>> > > > > > >>>>>> for >>>> > > > > > >>>>>>>> the provider will be installed as well - so we will >>>> be >>>> > able >>>> > > to >>>> > > > > get >>>> > > > > > >> rid >>>> > > > > > >>>>>> of >>>> > > > > > >>>>>>>> quite some code that is currently only used in >>>> airflow >>>> > > > > development >>>> > > > > > >>>>>>>> environment. >>>> > > > > > >>>>>>>> >>>> > > > > > >>>>>>>> J. >>>> > > > > > >>>>>>>> >>>> > > > > > >>>>>>>> >>>> > > > > > >>>>>>>> On Mon, Jan 6, 2025 at 11:41 PM Jarek Potiuk < >>>> > > ja...@potiuk.com> >>>> > > > > > >>>> wrote: >>>> > > > > > >>>>>>>> >>>> > > > > > >>>>>>>>> Those are very good questions :) >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>>>> On Mon, Jan 6, 2025 at 10:54 PM Ferruzzi, Dennis >>>> > > > > > >>>>>>>>> <ferru...@amazon.com.invalid> wrote: >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>>>>> To clarify that I understand your diagram >>>> correctly, >>>> > > let's say >>>> > > > > > you >>>> > > > > > >>>>>> clone >>>> > > > > > >>>>>>>>>> the Airflow repo to ~/workspace/airflow/. Does >>>> this >>>> > mean >>>> > > that >>>> > > > > > the >>>> > > > > > >>>>>> AWS Glue >>>> > > > > > >>>>>>>>>> Hook which used to live at >>>> > > > > > >>>>>>>>>> >>>> ~/workspace/airflow/providers/amazon/aws/hooks/glue.py >>>> > > (as a >>>> > > > > > >> random >>>> > > > > > >>>>>>>>>> example) will be located at >>>> > > > > > >>>>>>>>>> >>>> > > > > > >>>>>> >>>> > > > > > >>>> >>>> > > > > > >> >>>> > > > > > >>>> > > > > >>>> > > >>>> > >>>> ~/workspace/airflow/providers/amazon/aws/src/airflow/providers/amazon/aws/hooks/glue.py? >>>> > > > > > >>>>>>>>>> That feels unnecessarily repetitive to me, maybe it >>>> > makes >>>> > > > > sense >>>> > > > > > >> but >>>> > > > > > >>>>>> I'm >>>> > > > > > >>>>>>>>>> missing the context? >>>> > > > > > >>>>>>>>>> >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>>>> Yes - it means that there is this repetitiveness >>>> but for >>>> > a >>>> > > good >>>> > > > > > >>>>>> reason - >>>> > > > > > >>>>>>>>> those two "amazon/aws" serve different purpose: >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>>>> * The first "providers/amazon/aws" is just where the >>>> > whole >>>> > > > > > provider >>>> > > > > > >>>>>>>>> "complete project" is stored - it's basically a >>>> directory >>>> > > where >>>> > > > > > >> "aws >>>> > > > > > >>>>>>>>> provider" is stored, a convenient folder to locate >>>> it in, >>>> > > that >>>> > > > > > >> makes >>>> > > > > > >>>>>> it >>>> > > > > > >>>>>>>>> separate from other providers >>>> > > > > > >>>>>>>>> * The second "src/airflow/providers/amazon/aws" - >>>> is the >>>> > > python >>>> > > > > > >>>>>>>>> package where the source files is stored - this is >>>> how >>>> > > (inside >>>> > > > > > the >>>> > > > > > >>>>>>>>> sub-folder) you tell the actual python "import" to >>>> look >>>> > > for the >>>> > > > > > >>>>>> sources. >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>>>> .What really matters is that (eventually) >>>> > > > > > >>>>>>>>> `~/workspace/airflow/providers/amazon/aws/` can be >>>> > treated >>>> > > as a >>>> > > > > > >>>>>> completely >>>> > > > > > >>>>>>>>> separate python project - a source of a "standalone" >>>> > > provider >>>> > > > > > >> python >>>> > > > > > >>>>>>>>> project. >>>> > > > > > >>>>>>>>> There is a "pyproject.toml" file at the root of it >>>> and if >>>> > > you >>>> > > > > do >>>> > > > > > >> this >>>> > > > > > >>>>>> (for >>>> > > > > > >>>>>>>>> example): >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>>>> cd providers/amazon/aws/ >>>> > > > > > >>>>>>>>> uv sync >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>>>> And with it you will be able to work on AWS provider >>>> > > > > exclusively >>>> > > > > > >> as a >>>> > > > > > >>>>>>>>> separate project (this is not yet complete with the >>>> move >>>> > - >>>> > > > > tests >>>> > > > > > >> are >>>> > > > > > >>>>>> not >>>> > > > > > >>>>>>>>> entirely possible to run today - but it will be >>>> possible >>>> > as >>>> > > > > next >>>> > > > > > >> step >>>> > > > > > >>>>>> - I >>>> > > > > > >>>>>>>>> explained it in >>>> > > > > > >>>>>>>>> >>>> > > > > > >> >>>> > > >>>> https://github.com/apache/airflow/pull/45259#issuecomment-2572427916 >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>>>> This has a number of benefits, but one of them is >>>> that >>>> > you >>>> > > will >>>> > > > > > be >>>> > > > > > >>>>>> able to >>>> > > > > > >>>>>>>>> build provider by just running `build` command of >>>> your >>>> > > > > favourite >>>> > > > > > >>>>>>>>> PEP-standard compliant frontend: >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>>>> cd providers/amazon/aws/ >>>> > > > > > >>>>>>>>> `uv build` (or `hatch build` or `poetry build` or >>>> `flit >>>> > > build` >>>> > > > > > >> ).... >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>>>> This will create the provider package inside the >>>> `dist" >>>> > > > > folder. >>>> > > > > > I >>>> > > > > > >>>>>> just >>>> > > > > > >>>>>>>>> did it in my PR with `uv` in the first "airbyte` >>>> project: >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>>>> root@d74b3136d62f:/opt/airflow/providers/airbyte# >>>> uv >>>> > build >>>> > > > > > >>>>>>>>> Building source distribution... >>>> > > > > > >>>>>>>>> Building wheel from source distribution... >>>> > > > > > >>>>>>>>> Successfully built >>>> > > > > > >> dist/apache_airflow_providers_airbyte-5.0.0.tar.gz >>>> > > > > > >>>>>>>>> Successfully built >>>> > > > > > >>>>>>>>> >>>> > > dist/apache_airflow_providers_airbyte-5.0.0-py3-none-any.whl >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>>>> That's it. That also allows cases like installing >>>> > provider >>>> > > > > > packages >>>> > > > > > >>>>>> using >>>> > > > > > >>>>>>>>> git URLs - which I used earlier today to test if the >>>> > > incoming >>>> > > > > PR >>>> > > > > > of >>>> > > > > > >>>>>>>>> pygments is actually solving the problem we had >>>> yesteday >>>> > > > > > >>>>>>>>> https://github.com/apache/airflow/pull/45416 >>>> (basically >>>> > > we >>>> > > > > just >>>> > > > > > >>>>>> make our >>>> > > > > > >>>>>>>>> provider packages "standard" python packages that >>>> all the >>>> > > tools >>>> > > > > > >>>>>> support. >>>> > > > > > >>>>>>>>> Anyone who would like to install a commit, hash or >>>> branch >>>> > > > > version >>>> > > > > > >> of >>>> > > > > > >>>>>> the >>>> > > > > > >>>>>>>>> "airbyte" package from main version of Airflow repo >>>> will >>>> > be >>>> > > > > able >>>> > > > > > to >>>> > > > > > >>>>>> do: >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>>>> pip install "apache-airflow-providers-airbyte @ git+ >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>> https://github.com/apache/airflow.git/providers/airbyte@COMMIT_ID"; >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>>>> Currently in order to create the package we need to >>>> > > manually >>>> > > > > > >> extract >>>> > > > > > >>>>>> the >>>> > > > > > >>>>>>>>> "amazon" subtree, copy it elsewhere, prepare >>>> dynamically >>>> > > some >>>> > > > > > files >>>> > > > > > >>>>>>>>> (pyproject.toml, README.rst and few others) and >>>> only then >>>> > > we >>>> > > > > > build >>>> > > > > > >>>>>> the >>>> > > > > > >>>>>>>>> package. All this - copying file structure, >>>> creating new >>>> > > files, >>>> > > > > > >>>>>> running the >>>> > > > > > >>>>>>>>> build command after and finally deleting the copied >>>> files >>>> > > is >>>> > > > > now >>>> > > > > > - >>>> > > > > > >>>>>>>>> dynamically and under-the-hood created by "breeze >>>> > > > > > >> release-management >>>> > > > > > >>>>>>>>> prepare-provider-packages" command. With this >>>> change, the >>>> > > > > > directory >>>> > > > > > >>>>>>>>> structure in `git` repo of ours is totally standard >>>> and >>>> > > allows >>>> > > > > us >>>> > > > > > >>>> (and >>>> > > > > > >>>>>>>>> anyone else) to build the package directly from it. >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>>>> And what is the plan for system tests? As part of >>>> this >>>> > > > > > >>>>>> reorganization, >>>> > > > > > >>>>>>>>>> could they be moved into >>>> > > providers/{PROVIDER_ID}/tests/system? >>>> > > > > > >> That >>>> > > > > > >>>>>> seems >>>> > > > > > >>>>>>>>>> more intuitive to me than their current location in >>>> > > > > > >>>>>>>>>> >>>> providers/tests/system/{PROVIDER_ID}/example_foo.py. >>>> > > > > > >>>>>>>>>> >>>> > > > > > >>>>>>>>>> >>>> > > > > > >>>>>>>>> Oh yeah - I missed that in the original structure >>>> as the >>>> > > > > > "airbyte" >>>> > > > > > >>>>>>>>> provider (that I chose as first one) did not >>>> contain the >>>> > > > > "system" >>>> > > > > > >>>>>> tests - >>>> > > > > > >>>>>>>>> but one of the two providers after that i was >>>> planning to >>>> > > make >>>> > > > > > sure >>>> > > > > > >>>>>> system >>>> > > > > > >>>>>>>>> tests are covered. They are supposed to be moved to >>>> > > > > > "tests/system" >>>> > > > > > >> of >>>> > > > > > >>>>>>>>> course - Elad had similar question and I also >>>> explained >>>> > it >>>> > > in >>>> > > > > > >> detail >>>> > > > > > >>>>>> in >>>> > > > > > >>>>>>>>> >>>> > > > > > >> >>>> > > >>>> https://github.com/apache/airflow/pull/45259#issuecomment-2572427916 >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>>>> I hope it answers the questions. If not - I am >>>> happy to >>>> > add >>>> > > > > more >>>> > > > > > >>>>>>>>> clarifications :) >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>>>>> J. >>>> > > > > > >>>>>>>>>> >>>> > > > > > >>>>>>>>> >>>> > > > > > >>>>>>> >>>> > > > > > >>>>>>> >>>> > > > > > >>>>>> >>>> > > > > > >>>>>> >>>> > > > > > >>>> > --------------------------------------------------------------------- >>>> > > > > > >>>>>> To unsubscribe, e-mail: >>>> dev-unsubscr...@airflow.apache.org >>>> > > > > > >>>>>> For additional commands, e-mail: >>>> > dev-h...@airflow.apache.org >>>> > > > > > >>>>>> >>>> > > > > > >>>>>> >>>> > > > > > >>>> >>>> > > > > > >>>> >>>> > > > > > >>>> >>>> > > > > >>>> --------------------------------------------------------------------- >>>> > > > > > >>>> To unsubscribe, e-mail: >>>> dev-unsubscr...@airflow.apache.org >>>> > > > > > >>>> For additional commands, e-mail: >>>> dev-h...@airflow.apache.org >>>> > > > > > >>>> >>>> > > > > > >>>> >>>> > > > > > >> >>>> > > > > > >> >>>> > > > > > >> >>>> > > >>>> --------------------------------------------------------------------- >>>> > > > > > >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >>>> > > > > > >> For additional commands, e-mail: >>>> dev-h...@airflow.apache.org >>>> > > > > > >> >>>> > > > > > >> >>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> > --------------------------------------------------------------------- >>>> > > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >>>> > > > > > For additional commands, e-mail: dev-h...@airflow.apache.org >>>> > > > > > >>>> > > > > > >>>> > > > > >>>> > > > >>>> > > >>>> > > >>>> --------------------------------------------------------------------- >>>> > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >>>> > > For additional commands, e-mail: dev-h...@airflow.apache.org >>>> > > >>>> > > >>>> > >>>> >>>
