brunsgaard opened a new pull request, #10141: URL: https://github.com/apache/paimon/pull/10141
### Purpose `RESTTokenFileIO` refreshes a vended data token when less than one hour of its lifetime remains. The window is the constant `RESTApi.TOKEN_EXPIRATION_SAFE_TIME_MILLIS = 3_600_000L`. It entered as a literal in #5504 and became the constant in #5562. Neither PR gives a reason for one hour. One hour is also the lifetime most cloud STS services hand out by default: | Credential source | Default lifetime | Can the server ask for more? | | --- | --- | --- | | AWS STS `AssumeRole` | 3600 s | Yes, up to 12 h, if the role allows it | | Alibaba Cloud STS `AssumeRole` | 3600 s | Yes, up to the role's maximum | | GCP service account access token | 3600 s | Only with the org policy `iam.allowServiceAccountCredentialLifetimeExtension` | A REST server that vends these default tokens hands out tokens that never have more than one hour left, so the client refreshes on every file access. On a GCS-backed catalog we measured about 500 `loadTableToken` requests per minute from one Flink TaskManager. The token cache does nothing, and the catalog service and the STS endpoint take the load. Because the constant is inlined at compile time, a deployment cannot change it without a fork. This PR makes the window the catalog option `data-token.expiration-safe-time` and sets the default to 5 minutes. Iceberg's `VendedCredentialsProvider` refreshes vended S3 credentials 5 minutes before they expire, so Paimon and Iceberg clients now behave the same against a server that vends for both. With the new default a client refreshes once per token lifetime, and each access still has at least 5 minutes of credential left. A server that vends tokens shorter than 5 minutes can lower the option; a deployment that wants the old behaviour can set it to `1 h`. pypaimon gets the same option in `RESTTokenFileIO` and in the virtual filesystem (`PaimonRealStorage`), which both carried their own copy of the constant. ### Tests - `RESTTokenFileIOTest`: with the default window a token with 30 minutes left is loaded once across three accesses, and a token with 2 minutes left is reloaded on every access. A configured one hour window reloads a 30 minute token on every access; a configured 1 minute window loads a 2 minute token once. The window survives Java serialization. A negative window is rejected. - `RESTCatalogTest#testRefreshFileIOWhenExpired`: the first token now has 60 seconds left, inside the new default window, so the test still exercises a refresh. - pypaimon `rest_token_file_io_test.py`: the default window keeps a 30 minute token and expires a 2 minute token; a configured window overrides the default. ### API and Format New catalog option `data-token.expiration-safe-time` (duration, default `5 min`) in Java and pypaimon. `RESTApi.TOKEN_EXPIRATION_SAFE_TIME_MILLIS` is unchanged. No format change. ### Documentation One sentence in `docs/docs/concepts/rest/index.md` names the option and its default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
