yayaha opened a new pull request, #18818: URL: https://github.com/apache/iceberg/pull/18818
`S3FileIO` and `GCSFileIO` schedule a refresh of the credentials they hold (#15678, #15696), so that a FileIO serialized later carries valid credentials. A deserialized copy also schedules this refresh when it builds its storage clients. For example, Spark broadcasts the table to executors, and each executor deserializes its own copy of the FileIO. Each copy holds the same credential expiry as the original, so all open copies send a refresh request at the same time as the original. The scheduled refresh replaces only the held credentials. It does not update the storage clients that the FileIO already built, and the held credentials matter only when the FileIO is serialized again. So a copy does not use the result. A task that runs longer than its credentials still gets new credentials. When refresh is enabled, each storage client has its own credentials provider, `VendedCredentialsProvider` for S3 and `OAuth2CredentialsWithRefresh` for GCS. The provider fetches new credentials before the old ones expire. **This change does not change that path**. This change sets a transient flag in `initialize()`. Only an instance with the flag schedules the refresh. Java and Kryo deserialization do not call `initialize()`, so copies do not schedule it. This affects `GCSFileIO` when the catalog sends `gcs.oauth2.refresh-credentials-endpoint`. It affects `S3FileIO` when its scheduled refresh succeeds, for example when `credentials.uri` is set. With a REST catalog, that refresh fails today (#17810). #17912 fixes it, so `S3FileIO` with a REST catalog is affected once #17912 merges. A delegate that a deserialized `ResolvingFileIO` creates calls `initialize()`, so it still schedules a refresh. Related: #18817 keeps refreshed delegate credentials in `ResolvingFileIO`. Testing: added `deserializedCopyDoesNotScheduleRefresh` to `TestS3FileIOCredentialRefresh` and `TestGCSFileIOCredentialRefresh`. It checks Java and Kryo copies. --- **AI Disclosure** - Model: Claude Opus 5.5 - Platform/Tool: Claude Code - Human Oversight: Reviewed - Prompt Summary: Stop deserialized FileIO copies from scheduling the credential refresh, with tests. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
