eladkal commented on PR #29452: URL: https://github.com/apache/airflow/pull/29452#issuecomment-1426514058
> What would be the purpose of such generalisation? Out of all transfer operators only three of them operate entirely within AWS: DynamoDBToS3Operator RedshiftToS3Operator and S3ToRedshiftOperator. Another 15 have just one AWS connection, which is specified with aws_conn_id. >We could even go as far as generalising all transfer operators, as there is always an in and an out with them, but that sounds like a major revamp. What does everyone think? How can we make the most practical first step? But why? My initial thought was to generalize only the Aws to Aws transfer operators. There is no need to handle the non aws ones because there it's very clear who is the input conn and who is the output coon `HiveToDynamoDBOperator` - has `hiveserver2_conn_id`, `aws_conn_id` you know what is in/out without adding them to the parameter names. The issue we are discussing is relevant only for Aws to Aws transfers where it's not clear. We can have `source_aws_conn_id` and `dest_aws_conn_id` where the dest is default with the source value. Only when user override the default it will be the case of 2 distinct accounts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
