David-Banquet opened a new pull request, #4394:
URL: https://github.com/apache/iceberg-python/pull/4394

   Closes #2657.
   
   This follows @Fokko's proposal on #2657, PyArrow part only. Today 
`botocore_session` reaches the catalog clients but not the FileIO, which falls 
back to the default credential chain, so a misconfiguration only shows up on 
the first data file write.
   
   Changes in `PyArrowFileIO`:
   
   - When `botocore_session` is set and no `s3.access-key-id` / 
`client.access-key-id` is configured, the S3 filesystem gets the session's 
frozen credentials. Explicit properties keep precedence, which also covers 
credentials vended by a REST catalog.
   - The session is also skipped when `s3.role-arn` or `client.role-arn` / 
`client.assume-role.arn` is set, so the existing role assumption keeps working 
as before.
   - The frozen credentials are part of the `fs_by_scheme` cache key. Each 
`new_input` / `new_output` / `delete` calls `get_frozen_credentials()`, which 
refreshes them when they are about to expire, so a new token gets a new 
`S3FileSystem` and long-running processes keep working after the first token 
expires.
   - `BOTOCORE_SESSION` moves to `pyiceberg.io` so `pyarrow.py` can use it 
without importing `pyiceberg.catalog`; `pyiceberg.catalog` re-exports it.
   
   `FsspecFileIO` is left for a follow-up, as suggested. #4349 (docs) can still 
go in first; I'll adjust or drop its note once this one lands.
   
   Testing:
   
   - 2 unit tests in `tests/io/test_pyarrow.py`: the session credentials reach 
`S3FileSystem`, explicit ones win, and changing the session's credentials 
creates a second filesystem. Both fail on main.
   - `make test`: 4694 passed.
   - Against S3 Tables with a REST catalog and bogus `AWS_ACCESS_KEY_ID` in the 
environment (the repro from #2657): `append` fails on main with 
`INVALID_ACCESS_KEY_ID` in `CreateMultipartUpload`, and succeeds with this 
change.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to