ABoothInTheWild opened a new issue, #31137: URL: https://github.com/apache/airflow/issues/31137
### Apache Airflow version 2.6.0 ### What happened When accessing data in a "Requester Pays" bucket, the user's project needs to be supplied in the storage client's definition of the bucket, or set in the acl. When calling the "list" or "download" function from the GCSHook, there is no place to supply a user project id. This results in the following error: _Bucket is a requester pays bucket but no user project provided._ This is explicit in the GCP [documentation](https://cloud.google.com/storage/docs/using-requester-pays#storage-download-file-requester-pays-python). ### What you think should happen instead In the "insert_bucket_acl" function in the GCSHook, a user_project is optionally supplied for Requester Pays projects. This code looks like this: ``` """:param user_project: (Optional) The project to be billed for this request. Required for Requester Pays buckets.""" if user_project: bucket.acl.user_project = user_project bucket.acl.save() ``` I believe this code should be added to the list and download functions as well. This should also fix any operators from GCP to GCP/S3/Azure that want to transfer data from a "Requester Pays" bucket. ### How to reproduce Call hook.list() on any GCS bucket with Requester Pays enabled ************************************************************************* ```# list all files in an Google Cloud Storage bucket hook = GCSHook( gcp_conn_id=self.gcp_conn_id, delegate_to=self.delegate_to, impersonation_chain=self.google_impersonation_chain, ) self.log.info( 'Getting list of the files. Bucket: %s; Delimiter: %s; Prefix: %s', self.bucket, self.delimiter, self.prefix, ) files = hook.list(bucket_name=self.bucket, prefix=self.prefix, delimiter=self.delimiter)``` ### Operating System Debian 11 ### Versions of Apache Airflow Providers _No response_ ### Deployment Docker-Compose ### Deployment details _No response_ ### Anything else _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
