pankajastro commented on PR #30658: URL: https://github.com/apache/airflow/pull/30658#issuecomment-1515220848
> > > usually, we keep a transfer operator in Google provider if data move from some services to Google services like from Postgres to Bigquery or from s3 to Bigquery/GCS so not sure if Google provider is the right place for this transfer operator. > > > > > > I am new to AirFlow so I don't have an opinion on this. I added the module next to other similar operators (`BigQueryToMySqlOperator` and `BigQueryToMsSqlOperator`). Did you notice those operators? > > Summarizing: yes the current location is good. > > cc: @pankajastro : The rule is a bit more complex here and not only based on whether the provider is on the "To" or "From" side of the transfer. In most cases, yes, the "To" side is more important, but not for the reason you think. > > The reason is that usually the "To" side (means - service) is more interested in receiving the data and is more lilely to be maintained by whoever puts the focus on the "to" provider. But it only makes sense if you have provider like "S3 <=> GCS" - where you have Amazon on one side and Google on the other. In this case Google is more insterested to maintain `S3 To GCS` and Amazon is more interested in `GCS to S3` for obvious reasons. That's why the first one should be in Google provider and the other in the Amazon one - because respectively Google and Amazon will be interested in maintaining those. > > For Bigquery to MySQL or Postgres , this is not as easy - because there is no "service provider" who would be interested in getting data to Postgres / MySQL, because those are generic databases, and while they have soime "stakeholders" those stakeholders do not own the service that they would be interested to get data in. And in this case Google is determined to be the "more likely to maintain it". > > This is succintly described in our CONTRIBUTING.rst: > > > * It is often debatable where to put transfer operators but we agreed to the following criteria: > > > > * We use "maintainability" of the operators as the main criteria - so the transfer operator > > should be kept at the provider which has highest "interest" in the transfer operator > > * For Cloud Providers or Service providers that usually means that the transfer operators > > should land at the "target" side of the transfer thanks for the detailed explanation makes sense 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
