Fokko commented on issue #2657: URL: https://github.com/apache/iceberg-python/issues/2657#issuecomment-6808066385
I think we should fix this rather than only document it. Today `botocore_session` half works: the catalog calls go through the session, while the FileIO silently falls back to the default credential chain, so it only surfaces on the first write. Proposal: 1. When `botocore_session` is set and no explicit `s3.*` or `client.*` credentials are configured, the S3 FileIO uses the session's credentials. Explicit properties keep precedence, which also covers credentials vended by the REST catalog. 2. `PyArrowFileIO`: `S3FileSystem` only accepts static keys, so take `get_frozen_credentials()` from the session and include them in the `fs_by_scheme` cache key. botocore refreshes them when needed, so an expired token produces a new filesystem instead of failing after an hour. 3. `FsspecFileIO`: s3fs accepts a `session`, so we can pass it credentials that refresh. I'd start with PyArrow since it's the default FileIO, and do fsspec as a follow up. Happy to merge #4349 in the meantime. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
