potiuk commented on PR #30658:
URL: https://github.com/apache/airflow/pull/30658#issuecomment-1510437413
> > usually, we keep a transfer operator in Google provider if data move
from some services to Google services like from Postgres to Bigquery or from s3
to Bigquery/GCS so not sure if Google provider is the right place for this
transfer operator.
>
> I am new to AirFlow so I don't have an opinion on this. I added the module
next to other similar operators (`BigQueryToMySqlOperator` and
`BigQueryToMsSqlOperator`). Did you notice those operators?
Summarizing: yes the current location is good.
cc: @pankajastro : The rule is a bit more complex here and derived not only
based on whether the provider is on the "To" or "From" side of the transfer.
In most cases, yes, the "To" side is more important, but not for the reason you
think. The reason is that usually the "To" side (means - service) is more
interested in receiving the data and is more lilely to be maintained by whoever
puts the focus on the "to" provider. But it only makes sense if you have
provider like "S3 <=> GCS" - where you have Amazon on one side and Google on
the other. In this case Google is more insterested to maintain `S3 To GCS` and
Amazon is more interested in `GCS to S3` for obvious reasons. That's one the
first one should be in Google provider and the other in the Amazon one -
because respectively Google and Amazon will be interested in maintaining those.
For Bigquery to MySQL or Postgres , this is not as easy - because there is
no "service provider" who would be interested in getting data to Postgres /
MySQL (because those are generic databases, and while they have soime
"stakeholders" those stakeholders do not own the service that they would be
interested to get data in. And in this case Google is determined to be the
"more likely to maintain it".
This is succintly described in our CONTRIBUTING.rst:
> * It is often debatable where to put transfer operators but we agreed to
the following criteria:
> * We use "maintainability" of the operators as the main criteria - so
the transfer operator
should be kept at the provider which has highest "interest" in the
transfer operator
> * For Cloud Providers or Service providers that usually means that the
transfer operators
should land at the "target" side of the transfer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
