I agree with Jarek. Using the structure proposed by Kamil sounds really sensible to me.
Tomek On Fri, 4 Oct 2019 at 21:27, Jarek Potiuk <[email protected]> wrote: > Side comment: > > In the near future (AIP-26 ??) we can also adopt the structure reflected in > the current documentation restructuring by Kamil > > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html# > > > - > > Fundamentals > < > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#fundamentals > > > - > > ASF: Apache Software Foundation > < > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#asf-apache-software-foundation > > > - > > Service integrations > < > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#service-integrations > > > - > > Software integrations > < > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#software-integrations > > > - > > Protocol integrations > < > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#protocol-integrations > > > > Providers: > > - Azure: Microsoft Azure > < > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#azure-microsoft-azure > > > - AWS: Amazon Web Services > < > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#aws-amazon-web-services > > > - GCP: Google Cloud Platform > < > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#gcp-google-cloud-platform > > > > This will make it much cleaner on the airflow.* package level. > > J. > > On Fri, Oct 4, 2019 at 9:05 PM Driesprong, Fokko <[email protected]> > wrote: > > > I agree with Jarek's suggestion to add another module. Personally I don't > > like having too many on the airflow._ level, there are quite a few > already. > > > > Also, I would also go for providers. Mostly because we have hook*s* and > > operator*s*. When there are a lot of {operators,hooks,sensors} it makes > > sense to package these in a separate module. I could imagine that other > > companies that provide managed services > > < > > > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#service-integrations > > > > > might > > evolve to a separate package, but maybe we should come up with some > > criteria. > > > > Cheers, Fokko > > > > Op vr 4 okt. 2019 om 19:00 schreef Jarek Potiuk < > [email protected] > > >: > > > > > Not that I want to open Pandora's box :). But I think the longer name - > > the > > > worse. Provider is quite nice and short. > > > > > > I agree with plural 'providers' (as we have hooks and operators). So > for > > > consistency it should be indeed plural. > > > > > > J. > > > > > > pt., 4 paź 2019, 18:48 użytkownik Chris Palmer <[email protected]> > > > napisał: > > > > > > > My question about Oracle/MySql wasn't a serious one, but I forget > > > sometimes > > > > that sarcasm doesn't come across well on email. > > > > > > > > I guess my objection is that I don't think that 'provider' adds > > anything > > > of > > > > value. I'm not convinced that there needs to be a level between > > 'airflow' > > > > and 'google' but if going that route I would advocate for at least > the > > > > plural form 'providers' or the more descriptive 'cloud_providers'. > > > > > > > > Chris > > > > > > > > On Fri, Oct 4, 2019 at 11:57 AM Kamil Breguła < > > [email protected] > > > > > > > > wrote: > > > > > > > > > Hello, > > > > > > > > > > We think that only cloud providers should be separated from others, > > > > > because these services are integrated with each other. Very often, > > > > > when you use one cloud provider, you use many services of a given > > > > > provider. Using a single provider solution provides a uniform way > of > > > > > authorization, etc. A large number of problems and mechanisms are > > > > > common to one provider. You can see the amount of integration from > > > > > Microsoft, Azure, Google on the reference list. > > > > > > > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html > > > > > In this list, cloud reference providers have been placed in > separate > > > > > tables because they have a very large number of services. If Oracle > > > > > will have many integrations, it is worth emphasizing this fact and > > > > > moving these integrations to a separate place, so that it is easier > > to > > > > > find them. and use. Keeping all possible files in one place makes > it > > > > > very difficult to use them. > > > > > > > > > > Best regards, > > > > > > > > > > On Fri, Oct 4, 2019 at 5:32 PM Chris Palmer <[email protected]> > > > wrote: > > > > > > > > > > > > This seems unnecessary to me. > > > > > > > > > > > > Is everything going to be under some 'provider' or just certain > > sets > > > of > > > > > > operators, and if so what differentiates when something should be > > > > under a > > > > > > provider or not? For example, are the mysql operators going to go > > > under > > > > > > 'provider/oracle/'? > > > > > > > > > > > > Chris > > > > > > > > > > > > On Fri, Oct 4, 2019 at 9:21 AM Jarek Potiuk < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Agree with Ash. > > > > > > > > > > > > > > After doing the gcp move and seeing the result we agreed that > > > > > 'provider' is > > > > > > > better as additional prefix. > > > > > > > > > > > > > > If no-one objects (Lazy Consensus > > > > > > > <https://community.apache.org/committers/lazyConsensus.html>) > > till > > > > > Monday > > > > > > > 3.20 CEST, we will update AIP-21 and move the gcp operators to > > > > > > > *provider/google/[gcp,gsuite]*. > > > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > -- Tomasz Urbaszek Polidea <https://www.polidea.com/> | Junior Software Engineer M: +48 505 628 493 <+48505628493> E: [email protected] <[email protected]> Unique Tech Check out our projects! <https://www.polidea.com/our-work>
