anitakar commented on issue #15933: URL: https://github.com/apache/airflow/issues/15933#issuecomment-844556113
Having looked a little closer ( https://github.com/apache/airflow/tree/master/airflow/providers/google): 1. all the non cloud operators, sensors, hooks (ads, marketing_platform, firebase) could become their own modules (apache-airflow-providers-google-ads etc.) 2. all the files in cloud operators, sensors, hooks ( https://github.com/apache/airflow/tree/master/airflow/providers/google/cloud/operators) could become their own modules (apache-airflow-providers-google-dataflow etc.) 3. all transfer operators would have to move per service With modifying __init__.py in a smart way all operators could preserve the same import. The problem would be only with transfer operators as they would have to be moved not to have the same packages in different pypi modules and that would be a breaking change for the users. Seems like a lot of work. What would be the benefit? Being able to release for example firebase operators separately? śr., 19 maj 2021 o 22:56 Anita Fronczak ***@***.***> napisał(a): > Hey, > > To start the discussion. > > The split that would make logical sense to me would be per Cloud Service > (API), like shown be `gcloud services list`, with transfer operators > belonging to appropriate Google side package, like Google Cloud Storage for > gcs transfer operators. > > For the common part maybe an apache-airflow-providers-google-core package, > similarly to https://pypi.org/project/google-cloud-core/ ? > > The most problematic part would be keeping common python libraries (but > that could be solved by apache-airflow-providers-google-core package) and > other cloud provider libraries in sync, especially between transfer > operators, so > maybe apache-airflow-providers-google-transfer-othercloudprovider for > transfer operators in the end? > > That is what comes to my mind straight away but I am probably not seeing > some potential problems. WDYT > > Anita > > śr., 19 maj 2021 o 07:59 Jarek Potiuk ***@***.***> > napisał(a): > >> cc: @dstandish <https://github.com/dstandish> @kaxil >> <https://github.com/kaxil> @ashb <https://github.com/ashb> @mik-laj >> <https://github.com/mik-laj> @turbaszek <https://github.com/turbaszek> >> @anitakar <https://github.com/anitakar> >> >> — >> You are receiving this because you were mentioned. >> Reply to this email directly, view it on GitHub >> <https://github.com/apache/airflow/issues/15933#issuecomment-843770091>, >> or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/ABE6AZEVD5PX22PR6PB7YULTONHTDANCNFSM45D4T6YA> >> . >> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
