Oh, makes even more sense. I like the idea. T.
On Sat, May 30, 2020 at 3:14 PM Kamil Breguła <kamil.breg...@polidea.com> wrote: > I don't want to create package > airflow.providers.transfer > but package > airflow.providers.*.transfer > All classes will be in the same provider package but in a different nested > package. You will need to import packages from the transfer package, not > the operator package. > > > On Sat, May 30, 2020, 15:05 Tomasz Urbaszek <turbas...@apache.org> wrote: > > > I am in favor of this change. However, do I correctly understand that > > providers.transfer will need many other providers (GCS=gcp, S3=aws, > > etc)? > > > > T. > > > > > > On Sat, May 30, 2020 at 2:28 PM Kamil Breguła <kamil.breg...@polidea.com > > > > wrote: > > > > > > Hello, > > > > > > We currently have transfer operators and other operators in one > > > package - operators. > > > This causes some packages to be large and will grow all the time. For > > > example: airflow.providers.google.cloud.operators have 52 modules > > > For this reason, the idea arose to move these operators to the new > > package. > > > As a result, we will have the following provider package structure. > > > > > > airflow.providers.*.example_dags > > > airflow.providers.*.hooks > > > airflow.providers.*.operators > > > airflow.providers.*.secrets > > > airflow.providers.*.sensors > > > airflow.providers.*.transfer > > > airflow.providers.*.utils > > > > > > Such a division already exists in the documentation. We distinguish > > > operators, transfer operators, and sensors. > > > https://airflow.readthedocs.io/en/latest/_api/index.html#operators > > > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html > > > In my opinion, this change has many advantages: > > > * the documentation and the user habits would more suit the structure > > > of the code, > > > * would facilitate browsing of operators, > > > * would allow more comfortable documentation generation based on the > > > directory structure. It cannot be determined now whether > > > cloud_speech_to_text is transfer operators or a regular operator based > > > on the module name. > > > > > > The only minus is the need to change users' habits. Users will have to > > > start looking for operators in the new package, but if they find this > > > package, they will find the module faster. > > > > > > Best regards, > > > Kamil > > >
