eskarimov opened a new pull request #17446: URL: https://github.com/apache/airflow/pull/17446
Re-opened PR #17284 ### **Issue:** Currently Google Cloud Transfer Service operators `CloudDataTransferServiceGCSToGCSOperator` and `CloudDataTransferServiceS3ToGCSOperator` don't provide functionality to specify destination path, i.e. prefix for transferred objects, hence all the objects should simply land into the root of the bucket. ### **Solution:** [The Storage Transfer Service allows to specify prefix for transferred objects](https://cloud.google.com/storage-transfer/docs/reference/rest/v1/TransferSpec#GcsData), the PR adds this functionality with a new input parameter for operators above. By default it'd be `None`, so transferred objects will still land into the root of a bucket. Ideally I'd like to use just a single input parameter for specifying destination location, similar as it's done for [`S3ToGCSOperator`](https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/transfers/s3_to_gcs.py), parsing GCS URL, but it'd break backward compatibility for already existing pipelines. Hence, adding a new input parameter seems a more reasonable solution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
