stats-dev opened a new pull request, #2948:
URL: https://github.com/apache/iceberg-python/pull/2948
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
Closes #2841
# Rationale for this change
This PR adds explicit AWS profile support for both the Glue catalog client
and
fsspec-based S3 FileIO.
While `GlueCatalog` already supports profile configuration, fsspec-based S3
operations did not propagate profile selection to the underlying
`S3FileSystem` or async AWS session. As a result, users had to rely on
environment
variables or the default AWS profile, which makes it difficult to work with
multiple AWS configurations in parallel.
This change introduces two configuration properties:
- `client.profile-name`: a unified AWS profile for the catalog client and
FileIO
- `s3.profile-name`: an AWS profile specifically for S3 FileIO
Profile resolution follows this precedence:
1. `s3.profile-name`
2. `client.profile-name`
This ensures consistent and explicit credential selection across catalog and
FileIO layers when using the fsspec backend.
## Are these changes tested?
Yes. New unit tests were added to validate the profile propagation behavior.
- **Glue Catalog**
- Verifies that `boto3.Session(profile_name=...)` is created when
initializing
`GlueCatalog` with `client.profile-name`.
- **S3 FileIO (fsspec)**
- Verifies that `client.profile-name` or `s3.profile-name` results in the
creation of an async AWS session with the correct profile, which is then
passed to `S3FileSystem`.
The tests were run locally with:
```bash
pytest tests/catalog/test_glue_profile.py tests/io/test_fsspec_profile.py
```
Output would be:
```
==================== test session starts =====================
platform darwin -- Python 3.12.4, pytest-9.0.2, pluggy-1.6.0
rootdir: ${ROOTDIR}/iceberg-python
configfile: pyproject.toml
plugins: anyio-4.2.0, lazy-fixture-0.6.3, requests-mock-1.12.1
collected 3 items
tests/catalog/test_glue_profile.py . [ 33%]
tests/io/test_fsspec_profile.py .. [100%]
===================== 3 passed in 1.02s ======================
```
## Are there any user-facing changes?
Yes, this adds new configuration properties that users can set:
- `client.profile-name`: Sets the AWS profile for both the catalog client
and FileIO (unified configuration).
- `s3.profile-name`: Sets the AWS profile specifically for S3 FileIO.
**Example Usage:**
```python
catalog = GlueCatalog(
"my_catalog",
**{
"client.profile-name": "my-aws-profile",
# ... other config
}
)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
