haydn-j-evans opened a new pull request, #17389: URL: https://github.com/apache/iceberg/pull/17389
A common request is for Spark and Trino pods running on Kubernetes to commonly authenticate to REST catalogs (Lakekeeper, Polaris, etc.) using the pod's own service account token, mounted from a projected/auto-mounted file (typically /var/run/secrets/kubernetes.io/serviceaccount/token). Kubernetes rotates this token in place before it expires — but Iceberg's REST catalog OAuth2 config only supports a static token string set once at catalog creation. Once that token expires, there's no way to pick up the rotated value without restarting the process, which isn't practical for long-lived Spark/Trino/Kafka Connect deployments. This was requested in #13550 (Kafka Connect + Lakekeeper, mounting a K8s service-account token), which was auto-closed as stale without a fix — the underlying need (read a token from a file, with rotation) still applies today across any REST-catalog engine. What this adds A new token-path property, usable anywhere token is today, that points at a file instead of an inline string. The session automatically re-reads the file and swaps in the new token ahead of expiry — parsed from the token's JWT exp claim where available — by default 5 minutes before expiry, configurable via token-path-refresh-buffer-ms. On a transient read failure (e.g. racing Kubernetes' atomic file rotation), it retries on a short backoff rather than giving up. It reuses the existing OAuth2Util.AuthSession scheduled-refresh infrastructure (same shared thread pool, same cooperative-stop-on-close semantics as the existing HTTP-based token refresh), just swapping the refresh mechanism from "call the OAuth2 token endpoint" to "re-read the file" — no network calls are needed for the file-based refresh path. Since Spark, Trino, Kafka Connect, and other engines pass REST catalog properties straight through to RESTSessionCatalog as an unfiltered map, this requires no changes outside core — token-path is usable immediately via spark.sql.catalog.<name>.token-path=..., iceberg.catalog.token-path=... (Kafka Connect), or the equivalent connector property. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
