dabla opened a new issue, #76851: URL: https://github.com/apache/airflow/issues/76851
### Apache Airflow Provider(s) amazon ### Versions of Apache Airflow Providers apache-airflow-providers-amazon 9.40.0rc2 (same code on main and in 9.39.0) ### Apache Airflow version 3.4.0 ### What happened? `provide_bucket_name` fills in a missing `bucket_name` from different places depending on whether the decorated method is sync or async: - sync methods: from `self.service_config["bucket_name"]`, i.e. `extra.service_config.s3.bucket_name` of the AWS connection - async methods (coroutines and async generators): only from `connection.schema` https://github.com/apache/airflow/blob/main/providers/amazon/src/airflow/providers/amazon/aws/hooks/s3.py#L88-L127 So a connection that sets its bucket in `service_config`, which is what the sync methods read, gives `bucket_name=None` to every async method decorated with it: `get_head_object_async`, `get_file_metadata_async`, and since #75242 `aload_bytes`. botocore then fails in `validate_bucket_name`: ``` TypeError: expected string or bytes-like object, got 'NoneType' s3.py aload_bytes s3.py wrapper (provide_bucket_name) s3.py get_head_object_async aiobotocore/client.py _make_api_call botocore/handlers.py validate_bucket_name ``` `hook.load_bytes(data, key="k")` works on the same connection, but `await hook.aload_bytes(data, key="k")` does not. ### What you think should happen instead? Both branches should resolve the bucket the same way. The async branch could read `service_config` from the connection it already fetches with `get_async_connection()` (`connection.extra_dejson["service_config"]["s3"]["bucket_name"]`) instead of `self.service_config`, which would do a synchronous connection lookup inside the event loop. Whether `connection.schema` should stay as a fallback for async is open: the sync branch no longer uses it. ### How to reproduce ```python # AWS connection "aws_s3" with extra {"service_config": {"s3": {"bucket_name": "my-bucket"}}} from airflow.sdk import task @task async def upload(): from airflow.providers.amazon.aws.hooks.s3 import S3Hook hook = S3Hook(aws_conn_id="aws_s3") await hook.aload_bytes(b"data", key="some/key") # TypeError, bucket_name None ``` `S3Hook(aws_conn_id="aws_s3").load_bytes(b"data", key="some/key")` uploads to `my-bucket`. ### Anything else? Found while testing #75242 for the 9.40.0rc2 release (#76805). Happy to open the PR. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
