kuga14 opened a new issue #14610:
URL: https://github.com/apache/airflow/issues/14610
**Version: v2.0.1**
**Git Version:.release:2.0.1+beb8af5ac6c438c29e2c186145115fb1334a3735**
**Docker Image Tag 2.0.1-python3.8-build**
**What happened:**
Try to use GCSToS3Operator to sync data beetwen GCS and Yandex Object
Storage.
Getting error:
```
[2021-03-04 19:12:37,937] {taskinstance.py:1455} ERROR - An error occurred
(AccessDenied) when calling the PutObject operation: Access Denied
Traceback (most recent call last):
File
"/root/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line
1112, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File
"/root/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line
1285, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File
"/root/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line
1315, in _execute_task
result = task_copy.execute(context=context)
File
"/root/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/transfers/gcs_to_s3.py",
line 183, in execute
s3_hook.load_bytes(
File
"/root/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/hooks/s3.py",
line 61, in wrapper
return func(*bound_args.args, **bound_args.kwargs)
File
"/root/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/hooks/s3.py",
line 90, in wrapper
return func(*bound_args.args, **bound_args.kwargs)
File
"/root/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/hooks/s3.py",
line 600, in load_bytes
self._upload_file_obj(file_obj, key, bucket_name, replace, encrypt,
acl_policy)
File
"/root/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/hooks/s3.py",
line 654, in _upload_file_obj
client.upload_fileobj(file_obj, bucket_name, key, ExtraArgs=extra_args)
File "/root/.local/lib/python3.8/site-packages/boto3/s3/inject.py", line
539, in upload_fileobj
return future.result()
File "/root/.local/lib/python3.8/site-packages/s3transfer/futures.py",
line 106, in result
return self._coordinator.result()
File "/root/.local/lib/python3.8/site-packages/s3transfer/futures.py",
line 265, in result
raise self._exception
File "/root/.local/lib/python3.8/site-packages/s3transfer/tasks.py", line
126, in __call__
return self._execute_main(kwargs)
File "/root/.local/lib/python3.8/site-packages/s3transfer/tasks.py", line
150, in _execute_main
return_value = self._main(**kwargs)
File "/root/.local/lib/python3.8/site-packages/s3transfer/upload.py", line
692, in _main
client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
File "/root/.local/lib/python3.8/site-packages/botocore/client.py", line
357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/root/.local/lib/python3.8/site-packages/botocore/client.py", line
676, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when
calling the PutObject operation: Access Denied
```
GCSToS3Operator calls S3Hook.load_bytes function as:
```
s3_hook.load_bytes(
cast(bytes, file_bytes), key=dest_key,
replace=self.replace, acl_policy=self.s3_acl_policy
)
```
When I use S3Hook in PythonOperator to do the same thing I additionaly
specify bucket_name parameter:
```
s3_hook.load_bytes(
cast(bytes, file_bytes), key=dest_key,
bucket_name=bucket_name, replace=replace
)
```
and it works.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
