Well. Sensors are based on BaseSensorOperator which is based on BaseOperator. So they are technically also operators (just grand-children). In the future maybe we will have BaseTransferOperator as well (as per our discussion in Slack). I think this might make sense to have transfer at the same level as operator and sensor.
In many talks we already speak about those (Sensor/Transfer/Operators) as different kinds of beasts so having packages at the same level makes more sense IMHO.. J. . On Sat, May 30, 2020 at 5:54 PM Ash Berlin-Taylor <a...@apache.org> wrote: > > I understand what you mean, and some separation could make sense, but > transfer operators are still operators (unlike sensors which are a different > interface/base class). > > How would people feel about airflow.providers.*. operators.transfer.X? > > On 30 May 2020 15:36:42 BST, Felix Uellendall <felue...@pm.me.INVALID> wrote: > >I really like the idea, too. It makes the users search for transfers a > >lot easier. > > > >Best Regards, > >Felix > > > >Sent from ProtonMail Mobile > > > >On Sat, May 30, 2020 at 15:59, Jarek Potiuk <jarek.pot...@polidea.com> > >wrote: > > > >> I definitely like the idea - if we can discussion and voting on it > >before > >> we prepare backport package final (???) RC candidate, that would be > >great. > >> > >> On Sat, May 30, 2020 at 3:31 PM Tomasz Urbaszek > ><turbas...@apache.org> > >> wrote: > >> > >>> Oh, makes even more sense. I like the idea. > >>> > >>> T. > >>> > >>> On Sat, May 30, 2020 at 3:14 PM Kamil Breguła > ><kamil.breg...@polidea.com> > >>> wrote: > >>> > >>> > I don't want to create package > >>> > airflow.providers.transfer > >>> > but package > >>> > airflow.providers.*.transfer > >>> > All classes will be in the same provider package but in a > >different > >>> nested > >>> > package. You will need to import packages from the transfer > >package, not > >>> > the operator package. > >>> > > >>> > > >>> > On Sat, May 30, 2020, 15:05 Tomasz Urbaszek <turbas...@apache.org> > >>> wrote: > >>> > > >>> > > I am in favor of this change. However, do I correctly understand > >that > >>> > > providers.transfer will need many other providers (GCS=gcp, > >S3=aws, > >>> > > etc)? > >>> > > > >>> > > T. > >>> > > > >>> > > > >>> > > On Sat, May 30, 2020 at 2:28 PM Kamil Breguła < > >>> kamil.breg...@polidea.com > >>> > > > >>> > > wrote: > >>> > > > > >>> > > > Hello, > >>> > > > > >>> > > > We currently have transfer operators and other operators in > >one > >>> > > > package - operators. > >>> > > > This causes some packages to be large and will grow all the > >time. For > >>> > > > example: airflow.providers.google.cloud.operators have 52 > >modules > >>> > > > For this reason, the idea arose to move these operators to the > >new > >>> > > package. > >>> > > > As a result, we will have the following provider package > >structure. > >>> > > > > >>> > > > airflow.providers.*.example_dags > >>> > > > airflow.providers.*.hooks > >>> > > > airflow.providers.*.operators > >>> > > > airflow.providers.*.secrets > >>> > > > airflow.providers.*.sensors > >>> > > > airflow.providers.*.transfer > >>> > > > airflow.providers.*.utils > >>> > > > > >>> > > > Such a division already exists in the documentation. We > >distinguish > >>> > > > operators, transfer operators, and sensors. > >>> > > > > >https://airflow.readthedocs.io/en/latest/_api/index.html#operators > >>> > > > > >>> > >https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html > >>> > > > In my opinion, this change has many advantages: > >>> > > > * the documentation and the user habits would more suit the > >structure > >>> > > > of the code, > >>> > > > * would facilitate browsing of operators, > >>> > > > * would allow more comfortable documentation generation based > >on the > >>> > > > directory structure. It cannot be determined now whether > >>> > > > cloud_speech_to_text is transfer operators or a regular > >operator > >>> based > >>> > > > on the module name. > >>> > > > > >>> > > > The only minus is the need to change users' habits. Users will > >have > >>> to > >>> > > > start looking for operators in the new package, but if they > >find this > >>> > > > package, they will find the module faster. > >>> > > > > >>> > > > Best regards, > >>> > > > Kamil > >>> > > > >>> > > >>> > >> > >> -- > >> > >> Jarek Potiuk > >> Polidea <https://www.polidea.com/> | Principal Software Engineer > >> > >> M: +48 660 796 129 <+48660796129> > >> [image: Polidea] <https://www.polidea.com/> -- Jarek Potiuk Polidea | Principal Software Engineer M: +48 660 796 129
