Taragolis commented on code in PR #35273:
URL: https://github.com/apache/airflow/pull/35273#discussion_r1376726662
##########
airflow/providers/amazon/aws/fs/s3.py:
##########
@@ -85,7 +85,15 @@ def get_fs(conn_id: str | None) -> AbstractFileSystem:
if proxy_uri := s3_service_config.get(S3_PROXY_URI, None):
config_kwargs["proxies"] = {"http": proxy_uri, "https": proxy_uri}
- fs = S3FileSystem(session=session, config_kwargs=config_kwargs,
endpoint_url=endpoint_url)
+ anon = False
+ if (
+ aws.conn_config.aws_access_key_id is None
+ and aws.conn_config.aws_secret_access_key is None
+ and aws.conn_config.aws_session_token is None
+ ):
+ anon = True
Review Comment:
> There is no way to pass config_kwargs to this at the moment
But you use AWS Connection thought the hook, so users definitely could
provide `config_kwargs`
https://github.com/apache/airflow/blob/5fe65b7b1d205cf465e1ed2200bbe4d3893f145a/airflow/providers/amazon/aws/fs/s3.py#L58
I'm not sure the best way to resolve all cases, but seems like we need to
have explicit settings for this case in connection and documented it. Because
right now I can't see any word about `ObjectStoragePath("s3://xxxx") `into the
Amazon documentation, how to configure it, and how to use it. Without docs it
could be hard to triangle issues, answer on discussions in GH or Slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
