Fokko commented on code in PR #4394:
URL: https://github.com/apache/iceberg-python/pull/4394#discussion_r4600257839
##########
pyiceberg/io/pyarrow.py:
##########
@@ -513,7 +516,25 @@ def _initialize_oss_fs(self) -> FileSystem:
return S3FileSystem(**client_kwargs)
- def _initialize_s3_fs(self, netloc: str | None) -> FileSystem:
+ def _botocore_session_credentials(self) -> tuple[str, str, str | None] |
None:
+ """Return the frozen credentials of the botocore session when no
explicit S3 credentials or role are set."""
+ session: Any = self.properties.get(BOTOCORE_SESSION)
+ if session is None or get_first_property_value(self.properties,
S3_ACCESS_KEY_ID, AWS_ACCESS_KEY_ID) is not None:
+ return None
+ if get_first_property_value(self.properties, S3_ROLE_ARN,
AWS_ASSUME_ROLE_ARN, AWS_ROLE_ARN) is not None:
Review Comment:
This also needs to skip the session when `s3.anonymous` is set. Otherwise a
config with both `botocore_session` and `s3.anonymous=true` passes
`access_key`/`secret_key` together with `anonymous=True`, and PyArrow raises
`ValueError: Cannot pass anonymous=True together with access_key and
secret_key`. That config works on main today, so this would be a regression.
Something like:
```python
if property_as_bool(self.properties, S3_ANONYMOUS, False):
return None
```
A test case for it would be good too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]