bolkedebruin commented on code in PR #35273:
URL: https://github.com/apache/airflow/pull/35273#discussion_r1376848550
##########
airflow/providers/amazon/aws/fs/s3.py:
##########
@@ -85,7 +85,15 @@ def get_fs(conn_id: str | None) -> AbstractFileSystem:
if proxy_uri := s3_service_config.get(S3_PROXY_URI, None):
config_kwargs["proxies"] = {"http": proxy_uri, "https": proxy_uri}
- fs = S3FileSystem(session=session, config_kwargs=config_kwargs,
endpoint_url=endpoint_url)
+ anon = False
+ if (
+ aws.conn_config.aws_access_key_id is None
+ and aws.conn_config.aws_secret_access_key is None
+ and aws.conn_config.aws_session_token is None
+ ):
+ anon = True
Review Comment:
`config_kwargs` can be specified as part of the connection that is obtained
through the `conn_id`, which is how we do it everywhere else afaik? However, I
do want the default to fallback to anonymous when it cannot obtain credentials
either through `conn_id=None` or `conn_id=xxx`. This is the behavior of spark,
rasterio etc. This allows a bare
```python
path = ObjectStoragePath("s3://fmi-opendata-radar-geotiff") # <-- a real
bucket with anonymous access
```
To just work, without requiring extra settings.
I agree with you that this can be better documented, but it should be done
in the ObjectStorage documentation and not as part of a provider and as such I
think it is outside the scope of this PR.
I am happy to improve upon the documentation in a separate pr, but can also
use your assistance there due to your knowledge of this area.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
