Serpan07 opened a new issue, #17019: URL: https://github.com/apache/iceberg/issues/17019
### Apache Iceberg version 1.10.1 ### Query engine Kafka Connect ### Please describe the bug 🐞 ## Description ### Summary Apache Iceberg Java 1.10.1 does not propagate configured REST catalog custom headers (`header.*`) into the standalone `OAuth2Manager.tableSession(RESTClient, Map)` path. This causes credential vending to fail when Apache Polaris is configured with `require-header=true` for multi-tenant realm routing. The issue manifests when a long-running process (such as Kafka Connect Sink) holds a FileIO instance whose vended S3 credentials expire, triggering a credential refresh via the standalone table session path — which creates an HTTP session without the required custom headers. ### Environment | Component | Version | |-----------|---------| | Apache Iceberg (Java) | 1.10.1 | | Apache Iceberg Kafka Connect Sink | 1.10.1 | | Apache Polaris | 1.4.1 | | Auth | External OIDC (Keycloak) | | Storage | S3 (AWS and MinIO for reproduction) | | Credential vending | Polaris STS (`X-Iceberg-Access-Delegation: true`) | ### Configuration ```properties iceberg.catalog.type=rest iceberg.catalog.uri=http://polaris:8181/api/catalog iceberg.catalog.security=OAUTH2 iceberg.catalog.oauth2-server-uri=http://keycloak/realms/iceberg/protocol/openid-connect/token iceberg.catalog.credential=<client-id>:<client-secret> iceberg.catalog.token-exchange-enabled=false iceberg.catalog.header.Polaris-Realm=<tenant-realm> iceberg.catalog.header.X-Iceberg-Access-Delegation=true iceberg.catalog.fs.native-s3.enabled=true ``` Polaris configuration: ```properties polaris.realm-context.require-header=true ``` ### Expected behavior All HTTP calls to Polaris should include configured custom headers (`header.*`), including: - Catalog metadata calls (`GET /v1/config`, `GET /v1/.../tables/...`) - Credential vending / refresh calls triggered by the standalone FileIO path - Any call made by `OAuth2Manager.tableSession(RESTClient, Map)` ### Actual behavior When a FileIO needs to refresh its S3 credentials after they expire (e.g., after 900s with MinIO STS or 3600s with AWS STS), it invokes `OAuth2Manager.tableSession(RESTClient, Map<String, String> properties)`. This method creates an `AuthSession` using only `OAuth2Util.authHeaders(config.token())`, which contains only the `Authorization` header — not the custom `header.*` properties (like `Polaris-Realm`). The resulting HTTP call to Polaris is rejected with: ``` RESTException: Unable to process: Missing or invalid realm ``` The task crashes and enters a crash loop. ### Root cause In `OAuth2Manager.java` (line ~200 in 1.10.1): ```java public AuthSession tableSession(RESTClient sharedClient, Map<String, String> properties) { AuthConfig config = AuthConfig.fromProperties(properties); Map<String, String> headers = OAuth2Util.authHeaders(config.token()); // ↑ Only Authorization header, NOT the custom header.* properties OAuth2Util.AuthSession parent = new OAuth2Util.AuthSession(headers, config); ... } ``` The `properties` map passed to this method **does contain** the `header.Polaris-Realm` property, but it is never extracted via `RESTUtil.configHeaders(properties)` and merged into the session headers. Additionally, `newSessionFromTokenResponse` calls: ```java OAuth2Util.fetchToken(refreshClient, Map.of(), ...) // ↑ empty headers map ``` This means even token refresh calls made through this path will not include custom headers. ### Reproduction Minimal reproduction using Docker Compose (Kafka + Polaris + Keycloak + MinIO): 1. Configure Polaris with `require-header=true` and `STORAGE_CREDENTIAL_DURATION_SECONDS=900` 2. Configure Iceberg Kafka Connect Sink with: - `header.Polaris-Realm=local` - `X-Iceberg-Access-Delegation=true` - `token-exchange-enabled=false` - `fs.native-s3.enabled=true` - `commit.interval-ms=1200000` (20 min, longer than credential duration) 3. Start the connector and produce a message 4. Wait >15 min for S3 credentials to expire 5. The next commit attempt triggers a credential refresh via the standalone `tableSession` path 6. Polaris rejects the request: `Missing or invalid realm` 7. Task crashes Full reproduction environment: available on request. ### Who is affected Any long-running Iceberg client using: - REST catalog with custom `header.*` properties - Polaris multi-tenant with `require-header=true` - Credential vending (`X-Iceberg-Access-Delegation`) - A FileIO that outlives the vended credential duration This primarily affects **Kafka Connect Sink** (long-running, holds FileIO instances across commits) but could also affect long-running Spark jobs or any standalone component using `AuthManager.tableSession(RESTClient, Map)`. ### Suggested fix In `OAuth2Manager.tableSession(RESTClient, Map)`, merge custom headers from properties into the session: ```java public AuthSession tableSession(RESTClient sharedClient, Map<String, String> properties) { AuthConfig config = AuthConfig.fromProperties(properties); // Fix: include both auth headers AND custom headers from properties Map<String, String> customHeaders = RESTUtil.configHeaders(properties); Map<String, String> authHeaders = OAuth2Util.authHeaders(config.token()); Map<String, String> headers = RESTUtil.merge(customHeaders, authHeaders); OAuth2Util.AuthSession parent = new OAuth2Util.AuthSession(headers, config); ... } ``` Also ensure that `newSessionFromTokenResponse` passes the configured headers instead of `Map.of()`: ```java OAuth2Util.fetchToken(refreshClient, configuredHeaders, credential, scope, oauth2ServerUri, ...); ``` ### Willingness to contribute - [ ] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
