David-Banquet opened a new pull request, #4394: URL: https://github.com/apache/iceberg-python/pull/4394
Closes #2657. This follows @Fokko's proposal on #2657, PyArrow part only. Today `botocore_session` reaches the catalog clients but not the FileIO, which falls back to the default credential chain, so a misconfiguration only shows up on the first data file write. Changes in `PyArrowFileIO`: - When `botocore_session` is set and no `s3.access-key-id` / `client.access-key-id` is configured, the S3 filesystem gets the session's frozen credentials. Explicit properties keep precedence, which also covers credentials vended by a REST catalog. - The session is also skipped when `s3.role-arn` or `client.role-arn` / `client.assume-role.arn` is set, so the existing role assumption keeps working as before. - The frozen credentials are part of the `fs_by_scheme` cache key. Each `new_input` / `new_output` / `delete` calls `get_frozen_credentials()`, which refreshes them when they are about to expire, so a new token gets a new `S3FileSystem` and long-running processes keep working after the first token expires. - `BOTOCORE_SESSION` moves to `pyiceberg.io` so `pyarrow.py` can use it without importing `pyiceberg.catalog`; `pyiceberg.catalog` re-exports it. `FsspecFileIO` is left for a follow-up, as suggested. #4349 (docs) can still go in first; I'll adjust or drop its note once this one lands. Testing: - 2 unit tests in `tests/io/test_pyarrow.py`: the session credentials reach `S3FileSystem`, explicit ones win, and changing the session's credentials creates a second filesystem. Both fail on main. - `make test`: 4694 passed. - Against S3 Tables with a REST catalog and bogus `AWS_ACCESS_KEY_ID` in the environment (the repro from #2657): `append` fails on main with `INVALID_ACCESS_KEY_ID` in `CreateMultipartUpload`, and succeeds with this change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
