Ah and one comment (Pierre already encountered it) - before running `breeze start-airflow` you will need to rebuild the image. Breeze will nicely ask you to do so, so pressing `y` when it does will do the job.
This will happen more as we move the providers, so for a while if you see missing imports or similar - rebuild the image first. On Mon, Jan 20, 2025 at 6:32 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Update: > > We have now 3 providers moved: *airbyte, celery and edge - pre-release* > (thanks Jens for moving edge one!). I have already fixed a few initial > issues and improvements and added some consistency checks, there are two or > three more providers to move with some of the things I still need to add > (like devel deps) and we need to do a release cycle with providers to see > if everything works in the new structure, once we do it, we can accelerate > migration of the rest - I will prepare some scripts that will help with it. > > For now - one thing to remember: if you are adding/modifying dependencies > in the moved providers, they are no longer updated in *provider.yaml*, > you should edit them directly in respective `*pyproject.toml*`, For > example for celery, it will be `providers/celery/pyproject.toml` -> that > should work for required, optional and soon provider's devel dependencies > as well. Even if that file is automatically generated and some dependencies > (cross-provider ones) are added automatically if missing, any changes to > dependencies are preserved (including comments). I added some information > about it in our contributing docs: > https://github.com/apache/airflow/blob/main/contributing-docs/11_provider_packages.rst#where-providers-are-kept-in-our-repository > > Also apologies in advance to those who will have PRs in progress in > providers, once we move the provider, you will have to rebase your changes > and use the new location of provider src/ test/ doc/ dirs - this should be > rather painless, but I know it's annoying. That's why we also will migrate > provider-by-provider and spread it a bit to avoid big-bang and huge > disruption for everyone. > > J. > > > > On Tue, Jan 14, 2025 at 4:50 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> I will try to get celery next - we need a bugfix for it after yanking the >> last release. >> >> On Tue, Jan 14, 2025 at 9:46 AM Elad Kalif <elad...@apache.org> wrote: >> >>> I plan to cut a wave soon but we can't test on airbyte as there were no >>> changes since last release >>> >>> On Tue, Jan 14, 2025 at 8:52 AM Jarek Potiuk <ja...@potiuk.com> wrote: >>> >>> > First provider (airbyte) merged. Managed to walk around Sphinx without >>> > answering too many riddles finally. >>> > >>> > In the next coming days I will migrate a few more providers and once >>> we get >>> > through a next provider release cycle with those few providers and >>> fix/see >>> > any kind of teething issues I will add semi-automation to move the >>> provider >>> > one-by one and I will ask for help in individually migrating all of >>> them. >>> > >>> > The two limitations that will need to be solved later (so the provider >>> > projects are not yet truly standalone) >>> > >>> > * tests are only running from within airflow - you cannot run tests >>> inside >>> > `providers/airbyte" yet >>> > * doc build also currently works only from "airflow" main repo. - you >>> can't >>> > build docs as "command" in the build of the provider project >>> > >>> > Both are solvable but require some more reshuffling and sphinx scripts >>> > rewriting basically, so something that will be way easier when we >>> migrate >>> > all providers and remove the old provider approach. >>> > >>> > I will summarize the changes you should expect when contributing to a >>> > provider in a separate mail once I complete a few more providers. >>> > >>> > J. >>> > >>> > On Fri, Jan 10, 2025 at 5:52 PM Vincent Beck <vincb...@apache.org> >>> wrote: >>> > >>> > > Same, I am not strongly opinionated, just a preference :) >>> > > >>> > > On 2025/01/10 15:36:05 Vikram Koka wrote: >>> > > > I agree this makes sense. >>> > > > >>> > > > I was originally concerned that this would make it more difficult >>> to >>> > > ensure >>> > > > compatibility across providers for capabilities such as common.sql, >>> > > > objectstore, and so on. >>> > > > However, seeing that the "common" pattern would remain the same and >>> > it's >>> > > > only the code layout that is changing, and that we are getting rid >>> of a >>> > > ton >>> > > > of generated code, I am positive on this. >>> > > > >>> > > > >>> > > > On Fri, Jan 10, 2025 at 3:07 AM Jarek Potiuk <ja...@potiuk.com> >>> wrote: >>> > > > >>> > > > > So I propose letting the "doer" make the decision if we are >>> split. >>> > > > > >>> > > > > On Fri, Jan 10, 2025 at 11:52 AM Ash Berlin-Taylor < >>> a...@apache.org> >>> > > wrote: >>> > > > > >>> > > > > > Not strong at all, preference Is all. It sounds like Vincent >>> and I >>> > > are in >>> > > > > > the hyphen camp and you and Maciej are in the slash camp. >>> > > > > > >>> > > > > > +1 on the “I don’t care what code style is used as long as it >>> is >>> > > > > > programmatically enforced”. >>> > > > > > >>> > > > > > -a >>> > > > > > >>> > > > > > > On 10 Jan 2025, at 09:41, Jarek Potiuk <ja...@potiuk.com> >>> wrote: >>> > > > > > > >>> > > > > > > Is there anything else that "tastes" Ash ? A concrete reason >>> that >>> > > makes >>> > > > > > you >>> > > > > > > think the "-" prefix in this case is better than the "/" >>> folder? >>> > > How >>> > > > > > > strong is your "taste" preference and do you think it will >>> have >>> > > some >>> > > > > > > lasting effect if we choose to flatten the folder structure? >>> > > > > > > >>> > > > > > > I might make a small vote to see what is the preference of >>> people >>> > > if we >>> > > > > > > think this is an important aspect. >>> > > > > > > >>> > > > > > > BTW. This is why I really love black/ruff formatting - we >>> stopped >>> > > > > wasting >>> > > > > > > time on "taste" discussion - it does not matter what is the >>> > > individual >>> > > > > > > preference, consistency is more important and I prefer to do >>> > stuff >>> > > that >>> > > > > > > really matters but if people feel strongly that we should >>> discuss >>> > > it, I >>> > > > > > > might make a vote there. >>> > > > > > > >>> > > > > > > J. >>> > > > > > > >>> > > > > > > On Thu, Jan 9, 2025 at 5:48 PM Ash Berlin-Taylor < >>> a...@apache.org >>> > > >>> > > > > wrote: >>> > > > > > > >>> > > > > > >> My preference is for being “more direct” and not having >>> deeply >>> > > nested >>> > > > > > >> things where possible — I think Microsoft might be the one >>> case >>> > > where >>> > > > > > >> having extra folders makes sense. And I’m fine with things >>> not >>> > > being >>> > > > > > >> consistent across providers/groups of providers. >>> > > > > > >> >>> > > > > > >> -ash >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > > > > >>> On 8 Jan 2025, at 17:18, Jarek Potiuk <ja...@potiuk.com> >>> > wrote: >>> > > > > > >>> >>> > > > > > >>> Can you give an example of what might break why having >>> > > > > > >>> `providers/aapche-beam/src/airflow/providers/apache/beam`? >>> > > > > > >>> >>> > > > > > >>> Nothing will break. It's just: >>> > > > > > >>> >>> > > > > > >>> * the code will have to be a little more complex as it will >>> > have >>> > > to >>> > > > > do >>> > > > > > >> some >>> > > > > > >>> conditional writes of "-" "/" >>> > > > > > >>> * there will be inconsistency in the depth of folders - >>> outside >>> > > it >>> > > > > will >>> > > > > > >> be >>> > > > > > >>> 1, inside it will be 2 (as it is in your example)/ >>> > > > > > >>> * it will be a bit more convention/ complex to limit >>> related >>> > > > > providers >>> > > > > > >> (say >>> > > > > > >>> microsoft) - with the current scheme "providers/microsoft" >>> is >>> > the >>> > > > > > >> directory >>> > > > > > >>> containing all microsoft providers. If we change it to >>> "-", you >>> > > have >>> > > > > to >>> > > > > > >>> find all sub-directories following "microsoft-*" >>> convention. >>> > > > > > >>> >>> > > > > > >>> I am not super-strong on it - we could do either, it's >>> just my >>> > > > > > preference >>> > > > > > >>> to use folders for grouping related things (as folders were >>> > > designed >>> > > > > > >> for). >>> > > > > > >>> >>> > > > > > >>> J. >>> > > > > > >>> >>> > > > > > >>> On Wed, Jan 8, 2025 at 5:03 PM Ash Berlin-Taylor < >>> > a...@apache.org >>> > > > >>> > > > > > wrote: >>> > > > > > >>> >>> > > > > > >>>>> And we already have a number of mappings and conventions >>> to >>> > > handle >>> > > > > > >> that. >>> > > > > > >>>>> For example provider I'd mapping to dirs (apache.beam -> >>> > > > > > apache/beam), >>> > > > > > >>>> and >>> > > > > > >>>>> 'apache-airflow-providers-apache-beam' as package na e >>> and >>> > > > > > >>>>> airflow/providers/apache/beam as packages inside the >>> > > distribution. >>> > > > > > >> Those >>> > > > > > >>>>> will remain as they are - we cannot change them without >>> > > breaking >>> > > > > > >>>>> compatibility. >>> > > > > > >>>> >>> > > > > > >>>> Can you give an example of what might break why having >>> > > > > > >>>> `providers/aapche-beam/src/airflow/providers/apache/beam`? >>> > > > > > >>>> >>> > > > > > >>>> -a >>> > > > > > >>>> >>> > > > > > >>>>> On 7 Jan 2025, at 18:33, Jarek Potiuk <ja...@potiuk.com> >>> > > wrote: >>> > > > > > >>>>> >>> > > > > > >>>>> I think it will be better to keep it. >>> > > > > > >>>>> >>> > > > > > >>>>> The reason we have varying levels were to group things >>> > > together - >>> > > > > > >> mainly >>> > > > > > >>>>> Apache related providers, but also Microsoft. >>> > > > > > >>>>> >>> > > > > > >>>>> And we already have a number of mappings and conventions >>> to >>> > > handle >>> > > > > > >> that. >>> > > > > > >>>>> For example provider I'd mapping to dirs (apache.beam -> >>> > > > > > apache/beam), >>> > > > > > >>>> and >>> > > > > > >>>>> 'apache-airflow-providers-apache-beam' as package na e >>> and >>> > > > > > >>>>> airflow/providers/apache/beam as packages inside the >>> > > distribution. >>> > > > > > >> Those >>> > > > > > >>>>> will remain as they are - we cannot change them without >>> > > breaking >>> > > > > > >>>>> compatibility. >>> > > > > > >>>>> >>> > > > > > >>>>> So if we change it to a flat structure we will have some >>> > > > > > >> inconsistencies >>> > > > > > >>>> - >>> > > > > > >>>>> in some cases it will be single folder in others >>> (packages) >>> > > those >>> > > > > > will >>> > > > > > >> be >>> > > > > > >>>>> two folders. >>> > > > > > >>>>> >>> > > > > > >>>>> I think it will be more harm than good if we get rid of >>> the >>> > > > > 'folder' >>> > > > > > >>>>> structures - some of the code in breeze will have to >>> treat >>> > > those >>> > > > > > >>>>> differently as well. Nothing extraordinary and very >>> complex >>> > but >>> > > > > more >>> > > > > > >>>>> complex-ish than it should be - already on top of >>> handling >>> > > > > > potentially >>> > > > > > >>>>> nested folders >>> > > > > > >>>>> >>> > > > > > >>>>> So my preference would be to stay with apache/beam - it's >>> > just >>> > > more >>> > > > > > >>>>> consistently handling the case where provider packages >>> can be >>> > > > > > one-level >>> > > > > > >>>>> nested >>> > > > > > >>>>> >>> > > > > > >>>>> J >>> > > > > > >>>>> >>> > > > > > >>>>> >>> > > > > > >>>>> wt., 7 sty 2025, 19:00 użytkownik Vincent Beck < >>> > > > > vincb...@apache.org> >>> > > > > > >>>>> napisał: >>> > > > > > >>>>> >>> > > > > > >>>>>> Good question. I always found it confusing to have some >>> > > providers >>> > > > > at >>> > > > > > >>>>>> different level. Examples: >>> > > > > > >>>>>> - "airbyte" in "providers" directory (I would qualify >>> it as >>> > > > > > "regular" >>> > > > > > >>>>>> provider) >>> > > > > > >>>>>> - "hive" in "providers/apache" >>> > > > > > >>>>>> - "amazon" in "providers" but which contains only one >>> sub >>> > > > > directory >>> > > > > > >>>> "aws" >>> > > > > > >>>>>> >>> > > > > > >>>>>> I would be in favor of using "-" instead of "/" so that >>> all >>> > > > > > providers >>> > > > > > >>>> are >>> > > > > > >>>>>> at the same level. >>> > > > > > >>>>>> >>> > > > > > >>>>>> >>> > > > > > >>>>>> On 2025/01/07 16:38:10 Ash Berlin-Taylor wrote: >>> > > > > > >>>>>>> +1 one to this on general terms, it will hopefully >>> reduce a >>> > > lot >>> > > > > of >>> > > > > > >> the >>> > > > > > >>>>>> boilerplate we need. >>> > > > > > >>>>>>> >>> > > > > > >>>>>>> As for the amazon/aws example specifically that does >>> bring >>> > > up a >>> > > > > > >>>>>> question, should we have `/` or `-`.. to give some >>> examples: >>> > > > > > >>>>>>> >>> > > > > > >>>>>>> cncf kubernetes: ./providers/cncf/kubernetes or >>> > > > > > >>>>>> ./providers/cncf-kubernetes >>> > > > > > >>>>>>> Apache hive: ./providers/apache/hive or >>> > > ./providers/apache-hive >>> > > > > > >>>>>>> AWS: ./providers/amazon/aws or ./providers/amazon-aws >>> > > > > > >>>>>>> >>> > > > > > >>>>>>> There is no requirement from python etc on one form or >>> the >>> > > other >>> > > > > > (as >>> > > > > > >>>>>> it’s just a folder, not part of the module name), so >>> it’s >>> > what >>> > > > > ever >>> > > > > > >>>> makes >>> > > > > > >>>>>> most sense to us. >>> > > > > > >>>>>>> >>> > > > > > >>>>>>> Jarek and Dennis (and others): what are your >>> preferences on >>> > > these >>> > > > > > >>>> styles? >>> > > > > > >>>>>>> >>> > > > > > >>>>>>> -ash >>> > > > > > >>>>>>> >>> > > > > > >>>>>>>> On 6 Jan 2025, at 22:51, Jarek Potiuk < >>> ja...@potiuk.com> >>> > > wrote: >>> > > > > > >>>>>>>> >>> > > > > > >>>>>>>> Oh. . And one other benefit of it will be that we >>> will be >>> > > able >>> > > > > to >>> > > > > > >> get >>> > > > > > >>>>>> rid >>> > > > > > >>>>>>>> of about 40% of the "Providers Manager" code. >>> Currently, >>> > in >>> > > > > > >> Providers >>> > > > > > >>>>>>>> manager we have a lot of "ifs" that make it possible >>> to >>> > use >>> > > > > > >> providers >>> > > > > > >>>>>> in >>> > > > > > >>>>>>>> breeze and local environment from the sources. In >>> > > "production" >>> > > > > > >>>>>> installation >>> > > > > > >>>>>>>> we are using "get_provider_info" entry points to >>> discover >>> > > > > > providers >>> > > > > > >>>>>> and >>> > > > > > >>>>>>>> discover if provider is installed, but when you use >>> > current >>> > > > > > >> providers >>> > > > > > >>>>>>>> installed in Breeze to inside "airflow", we rely on >>> > > > > > `provider.yaml` >>> > > > > > >> to >>> > > > > > >>>>>> be >>> > > > > > >>>>>>>> present in the "airflow.providers.PROVIDER_ID" path - >>> so >>> > we >>> > > > > > >>>> effectively >>> > > > > > >>>>>>>> have two paths of discovering information about the >>> > > providers >>> > > > > > >>>>>> installed. >>> > > > > > >>>>>>>> >>> > > > > > >>>>>>>> After all providers are migrated to the new >>> structure, all >>> > > > > > providers >>> > > > > > >>>>>> are >>> > > > > > >>>>>>>> separate "distributions" - and when you run `uv sync` >>> > > (which >>> > > > > will >>> > > > > > >>>>>> install >>> > > > > > >>>>>>>> all providers thanks to workspace feature) or `pip >>> install >>> > > -e >>> > > > > > >>>>>>>> ./providers/aws` (which you will have to do manually >>> to >>> > > work on >>> > > > > > the >>> > > > > > >>>>>>>> provider - if you use `pip` rather than uv) - then we >>> will >>> > > not >>> > > > > > have >>> > > > > > >> to >>> > > > > > >>>>>> use >>> > > > > > >>>>>>>> the separate path to read provider.yaml, because the >>> right >>> > > > > > >> entrypoint >>> > > > > > >>>>>> for >>> > > > > > >>>>>>>> the provider will be installed as well - so we will be >>> > able >>> > > to >>> > > > > get >>> > > > > > >> rid >>> > > > > > >>>>>> of >>> > > > > > >>>>>>>> quite some code that is currently only used in airflow >>> > > > > development >>> > > > > > >>>>>>>> environment. >>> > > > > > >>>>>>>> >>> > > > > > >>>>>>>> J. >>> > > > > > >>>>>>>> >>> > > > > > >>>>>>>> >>> > > > > > >>>>>>>> On Mon, Jan 6, 2025 at 11:41 PM Jarek Potiuk < >>> > > ja...@potiuk.com> >>> > > > > > >>>> wrote: >>> > > > > > >>>>>>>> >>> > > > > > >>>>>>>>> Those are very good questions :) >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>>>> On Mon, Jan 6, 2025 at 10:54 PM Ferruzzi, Dennis >>> > > > > > >>>>>>>>> <ferru...@amazon.com.invalid> wrote: >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>>>>> To clarify that I understand your diagram correctly, >>> > > let's say >>> > > > > > you >>> > > > > > >>>>>> clone >>> > > > > > >>>>>>>>>> the Airflow repo to ~/workspace/airflow/. Does this >>> > mean >>> > > that >>> > > > > > the >>> > > > > > >>>>>> AWS Glue >>> > > > > > >>>>>>>>>> Hook which used to live at >>> > > > > > >>>>>>>>>> >>> ~/workspace/airflow/providers/amazon/aws/hooks/glue.py >>> > > (as a >>> > > > > > >> random >>> > > > > > >>>>>>>>>> example) will be located at >>> > > > > > >>>>>>>>>> >>> > > > > > >>>>>> >>> > > > > > >>>> >>> > > > > > >> >>> > > > > > >>> > > > > >>> > > >>> > >>> ~/workspace/airflow/providers/amazon/aws/src/airflow/providers/amazon/aws/hooks/glue.py? >>> > > > > > >>>>>>>>>> That feels unnecessarily repetitive to me, maybe it >>> > makes >>> > > > > sense >>> > > > > > >> but >>> > > > > > >>>>>> I'm >>> > > > > > >>>>>>>>>> missing the context? >>> > > > > > >>>>>>>>>> >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>>>> Yes - it means that there is this repetitiveness but >>> for >>> > a >>> > > good >>> > > > > > >>>>>> reason - >>> > > > > > >>>>>>>>> those two "amazon/aws" serve different purpose: >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>>>> * The first "providers/amazon/aws" is just where the >>> > whole >>> > > > > > provider >>> > > > > > >>>>>>>>> "complete project" is stored - it's basically a >>> directory >>> > > where >>> > > > > > >> "aws >>> > > > > > >>>>>>>>> provider" is stored, a convenient folder to locate >>> it in, >>> > > that >>> > > > > > >> makes >>> > > > > > >>>>>> it >>> > > > > > >>>>>>>>> separate from other providers >>> > > > > > >>>>>>>>> * The second "src/airflow/providers/amazon/aws" - is >>> the >>> > > python >>> > > > > > >>>>>>>>> package where the source files is stored - this is >>> how >>> > > (inside >>> > > > > > the >>> > > > > > >>>>>>>>> sub-folder) you tell the actual python "import" to >>> look >>> > > for the >>> > > > > > >>>>>> sources. >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>>>> .What really matters is that (eventually) >>> > > > > > >>>>>>>>> `~/workspace/airflow/providers/amazon/aws/` can be >>> > treated >>> > > as a >>> > > > > > >>>>>> completely >>> > > > > > >>>>>>>>> separate python project - a source of a "standalone" >>> > > provider >>> > > > > > >> python >>> > > > > > >>>>>>>>> project. >>> > > > > > >>>>>>>>> There is a "pyproject.toml" file at the root of it >>> and if >>> > > you >>> > > > > do >>> > > > > > >> this >>> > > > > > >>>>>> (for >>> > > > > > >>>>>>>>> example): >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>>>> cd providers/amazon/aws/ >>> > > > > > >>>>>>>>> uv sync >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>>>> And with it you will be able to work on AWS provider >>> > > > > exclusively >>> > > > > > >> as a >>> > > > > > >>>>>>>>> separate project (this is not yet complete with the >>> move >>> > - >>> > > > > tests >>> > > > > > >> are >>> > > > > > >>>>>> not >>> > > > > > >>>>>>>>> entirely possible to run today - but it will be >>> possible >>> > as >>> > > > > next >>> > > > > > >> step >>> > > > > > >>>>>> - I >>> > > > > > >>>>>>>>> explained it in >>> > > > > > >>>>>>>>> >>> > > > > > >> >>> > > https://github.com/apache/airflow/pull/45259#issuecomment-2572427916 >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>>>> This has a number of benefits, but one of them is >>> that >>> > you >>> > > will >>> > > > > > be >>> > > > > > >>>>>> able to >>> > > > > > >>>>>>>>> build provider by just running `build` command of >>> your >>> > > > > favourite >>> > > > > > >>>>>>>>> PEP-standard compliant frontend: >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>>>> cd providers/amazon/aws/ >>> > > > > > >>>>>>>>> `uv build` (or `hatch build` or `poetry build` or >>> `flit >>> > > build` >>> > > > > > >> ).... >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>>>> This will create the provider package inside the >>> `dist" >>> > > > > folder. >>> > > > > > I >>> > > > > > >>>>>> just >>> > > > > > >>>>>>>>> did it in my PR with `uv` in the first "airbyte` >>> project: >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>>>> root@d74b3136d62f:/opt/airflow/providers/airbyte# uv >>> > build >>> > > > > > >>>>>>>>> Building source distribution... >>> > > > > > >>>>>>>>> Building wheel from source distribution... >>> > > > > > >>>>>>>>> Successfully built >>> > > > > > >> dist/apache_airflow_providers_airbyte-5.0.0.tar.gz >>> > > > > > >>>>>>>>> Successfully built >>> > > > > > >>>>>>>>> >>> > > dist/apache_airflow_providers_airbyte-5.0.0-py3-none-any.whl >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>>>> That's it. That also allows cases like installing >>> > provider >>> > > > > > packages >>> > > > > > >>>>>> using >>> > > > > > >>>>>>>>> git URLs - which I used earlier today to test if the >>> > > incoming >>> > > > > PR >>> > > > > > of >>> > > > > > >>>>>>>>> pygments is actually solving the problem we had >>> yesteday >>> > > > > > >>>>>>>>> https://github.com/apache/airflow/pull/45416 >>> (basically >>> > > we >>> > > > > just >>> > > > > > >>>>>> make our >>> > > > > > >>>>>>>>> provider packages "standard" python packages that >>> all the >>> > > tools >>> > > > > > >>>>>> support. >>> > > > > > >>>>>>>>> Anyone who would like to install a commit, hash or >>> branch >>> > > > > version >>> > > > > > >> of >>> > > > > > >>>>>> the >>> > > > > > >>>>>>>>> "airbyte" package from main version of Airflow repo >>> will >>> > be >>> > > > > able >>> > > > > > to >>> > > > > > >>>>>> do: >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>>>> pip install "apache-airflow-providers-airbyte @ git+ >>> > > > > > >>>>>>>>> >>> > > > > > >>> https://github.com/apache/airflow.git/providers/airbyte@COMMIT_ID"; >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>>>> Currently in order to create the package we need to >>> > > manually >>> > > > > > >> extract >>> > > > > > >>>>>> the >>> > > > > > >>>>>>>>> "amazon" subtree, copy it elsewhere, prepare >>> dynamically >>> > > some >>> > > > > > files >>> > > > > > >>>>>>>>> (pyproject.toml, README.rst and few others) and only >>> then >>> > > we >>> > > > > > build >>> > > > > > >>>>>> the >>> > > > > > >>>>>>>>> package. All this - copying file structure, creating >>> new >>> > > files, >>> > > > > > >>>>>> running the >>> > > > > > >>>>>>>>> build command after and finally deleting the copied >>> files >>> > > is >>> > > > > now >>> > > > > > - >>> > > > > > >>>>>>>>> dynamically and under-the-hood created by "breeze >>> > > > > > >> release-management >>> > > > > > >>>>>>>>> prepare-provider-packages" command. With this >>> change, the >>> > > > > > directory >>> > > > > > >>>>>>>>> structure in `git` repo of ours is totally standard >>> and >>> > > allows >>> > > > > us >>> > > > > > >>>> (and >>> > > > > > >>>>>>>>> anyone else) to build the package directly from it. >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>>>> And what is the plan for system tests? As part of >>> this >>> > > > > > >>>>>> reorganization, >>> > > > > > >>>>>>>>>> could they be moved into >>> > > providers/{PROVIDER_ID}/tests/system? >>> > > > > > >> That >>> > > > > > >>>>>> seems >>> > > > > > >>>>>>>>>> more intuitive to me than their current location in >>> > > > > > >>>>>>>>>> providers/tests/system/{PROVIDER_ID}/example_foo.py. >>> > > > > > >>>>>>>>>> >>> > > > > > >>>>>>>>>> >>> > > > > > >>>>>>>>> Oh yeah - I missed that in the original structure as >>> the >>> > > > > > "airbyte" >>> > > > > > >>>>>>>>> provider (that I chose as first one) did not contain >>> the >>> > > > > "system" >>> > > > > > >>>>>> tests - >>> > > > > > >>>>>>>>> but one of the two providers after that i was >>> planning to >>> > > make >>> > > > > > sure >>> > > > > > >>>>>> system >>> > > > > > >>>>>>>>> tests are covered. They are supposed to be moved to >>> > > > > > "tests/system" >>> > > > > > >> of >>> > > > > > >>>>>>>>> course - Elad had similar question and I also >>> explained >>> > it >>> > > in >>> > > > > > >> detail >>> > > > > > >>>>>> in >>> > > > > > >>>>>>>>> >>> > > > > > >> >>> > > https://github.com/apache/airflow/pull/45259#issuecomment-2572427916 >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>>>> I hope it answers the questions. If not - I am happy >>> to >>> > add >>> > > > > more >>> > > > > > >>>>>>>>> clarifications :) >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>>>>> J. >>> > > > > > >>>>>>>>>> >>> > > > > > >>>>>>>>> >>> > > > > > >>>>>>> >>> > > > > > >>>>>>> >>> > > > > > >>>>>> >>> > > > > > >>>>>> >>> > > > > > >>> > --------------------------------------------------------------------- >>> > > > > > >>>>>> To unsubscribe, e-mail: >>> dev-unsubscr...@airflow.apache.org >>> > > > > > >>>>>> For additional commands, e-mail: >>> > dev-h...@airflow.apache.org >>> > > > > > >>>>>> >>> > > > > > >>>>>> >>> > > > > > >>>> >>> > > > > > >>>> >>> > > > > > >>>> >>> > > > > >>> --------------------------------------------------------------------- >>> > > > > > >>>> To unsubscribe, e-mail: >>> dev-unsubscr...@airflow.apache.org >>> > > > > > >>>> For additional commands, e-mail: >>> dev-h...@airflow.apache.org >>> > > > > > >>>> >>> > > > > > >>>> >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> >>> > > --------------------------------------------------------------------- >>> > > > > > >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >>> > > > > > >> For additional commands, e-mail: >>> dev-h...@airflow.apache.org >>> > > > > > >> >>> > > > > > >> >>> > > > > > >>> > > > > > >>> > > > > > >>> > --------------------------------------------------------------------- >>> > > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >>> > > > > > For additional commands, e-mail: dev-h...@airflow.apache.org >>> > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > > --------------------------------------------------------------------- >>> > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >>> > > For additional commands, e-mail: dev-h...@airflow.apache.org >>> > > >>> > > >>> > >>> >>
