yayaha opened a new pull request, #18818:
URL: https://github.com/apache/iceberg/pull/18818

   `S3FileIO` and `GCSFileIO` schedule a refresh of the credentials they hold 
(#15678, #15696), so that a FileIO serialized later carries valid credentials. 
A deserialized copy also schedules this refresh when it builds its storage 
clients.
   
   For example, Spark broadcasts the table to executors, and each executor 
deserializes its own copy of the FileIO. Each copy holds the same credential 
expiry as the original, so all open copies send a refresh request at the same 
time as the original. The scheduled refresh replaces only the held credentials. 
It does not update the storage clients that the FileIO already built, and the 
held credentials matter only when the FileIO is serialized again. So a copy 
does not use the result.
   
   A task that runs longer than its credentials still gets new credentials. 
When refresh is enabled, each storage client has its own credentials provider, 
`VendedCredentialsProvider` for S3 and `OAuth2CredentialsWithRefresh` for GCS. 
The provider fetches new credentials before the old ones expire. **This change 
does not change that path**.
   
   This change sets a transient flag in `initialize()`. Only an instance with 
the flag schedules the refresh. Java and Kryo deserialization do not call 
`initialize()`, so copies do not schedule it.
   
   This affects `GCSFileIO` when the catalog sends 
`gcs.oauth2.refresh-credentials-endpoint`. It affects `S3FileIO` when its 
scheduled refresh succeeds, for example when `credentials.uri` is set. With a 
REST catalog, that refresh fails today (#17810). #17912 fixes it, so `S3FileIO` 
with a REST catalog is affected once #17912 merges.
   
   A delegate that a deserialized `ResolvingFileIO` creates calls 
`initialize()`, so it still schedules a refresh.
   
   Related: #18817 keeps refreshed delegate credentials in `ResolvingFileIO`.
   
   Testing: added `deserializedCopyDoesNotScheduleRefresh` to 
`TestS3FileIOCredentialRefresh` and `TestGCSFileIOCredentialRefresh`. It checks 
Java and Kryo copies.
   
   ---
   **AI Disclosure**
   - Model: Claude Opus 5.5
   - Platform/Tool: Claude Code
   - Human Oversight: Reviewed
   - Prompt Summary: Stop deserialized FileIO copies from scheduling the 
credential refresh, with tests.
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to