I understand what you mean, and some separation could make sense, but transfer operators are still operators (unlike sensors which are a different interface/base class).
How would people feel about airflow.providers.*. operators.transfer.X? On 30 May 2020 15:36:42 BST, Felix Uellendall <felue...@pm.me.INVALID> wrote: >I really like the idea, too. It makes the users search for transfers a >lot easier. > >Best Regards, >Felix > >Sent from ProtonMail Mobile > >On Sat, May 30, 2020 at 15:59, Jarek Potiuk <jarek.pot...@polidea.com> >wrote: > >> I definitely like the idea - if we can discussion and voting on it >before >> we prepare backport package final (???) RC candidate, that would be >great. >> >> On Sat, May 30, 2020 at 3:31 PM Tomasz Urbaszek ><turbas...@apache.org> >> wrote: >> >>> Oh, makes even more sense. I like the idea. >>> >>> T. >>> >>> On Sat, May 30, 2020 at 3:14 PM Kamil Breguła ><kamil.breg...@polidea.com> >>> wrote: >>> >>> > I don't want to create package >>> > airflow.providers.transfer >>> > but package >>> > airflow.providers.*.transfer >>> > All classes will be in the same provider package but in a >different >>> nested >>> > package. You will need to import packages from the transfer >package, not >>> > the operator package. >>> > >>> > >>> > On Sat, May 30, 2020, 15:05 Tomasz Urbaszek <turbas...@apache.org> >>> wrote: >>> > >>> > > I am in favor of this change. However, do I correctly understand >that >>> > > providers.transfer will need many other providers (GCS=gcp, >S3=aws, >>> > > etc)? >>> > > >>> > > T. >>> > > >>> > > >>> > > On Sat, May 30, 2020 at 2:28 PM Kamil Breguła < >>> kamil.breg...@polidea.com >>> > > >>> > > wrote: >>> > > > >>> > > > Hello, >>> > > > >>> > > > We currently have transfer operators and other operators in >one >>> > > > package - operators. >>> > > > This causes some packages to be large and will grow all the >time. For >>> > > > example: airflow.providers.google.cloud.operators have 52 >modules >>> > > > For this reason, the idea arose to move these operators to the >new >>> > > package. >>> > > > As a result, we will have the following provider package >structure. >>> > > > >>> > > > airflow.providers.*.example_dags >>> > > > airflow.providers.*.hooks >>> > > > airflow.providers.*.operators >>> > > > airflow.providers.*.secrets >>> > > > airflow.providers.*.sensors >>> > > > airflow.providers.*.transfer >>> > > > airflow.providers.*.utils >>> > > > >>> > > > Such a division already exists in the documentation. We >distinguish >>> > > > operators, transfer operators, and sensors. >>> > > > >https://airflow.readthedocs.io/en/latest/_api/index.html#operators >>> > > > >>> >https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html >>> > > > In my opinion, this change has many advantages: >>> > > > * the documentation and the user habits would more suit the >structure >>> > > > of the code, >>> > > > * would facilitate browsing of operators, >>> > > > * would allow more comfortable documentation generation based >on the >>> > > > directory structure. It cannot be determined now whether >>> > > > cloud_speech_to_text is transfer operators or a regular >operator >>> based >>> > > > on the module name. >>> > > > >>> > > > The only minus is the need to change users' habits. Users will >have >>> to >>> > > > start looking for operators in the new package, but if they >find this >>> > > > package, they will find the module faster. >>> > > > >>> > > > Best regards, >>> > > > Kamil >>> > > >>> > >>> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/>
