Serpan07 opened a new issue, #17810:
URL: https://github.com/apache/iceberg/issues/17810
## Apache Iceberg version
1.11.0
## Query engine
Spark 4.0
## Please describe the bug 🐞
When using a REST catalog (Apache Polaris) with
`X-Iceberg-Access-Delegation: vended-credentials`, the new
`S3FileIO.refreshStorageCredentials()` method introduced in PR #15678 fails
with:
```
WARN S3FileIO: Failed to refresh storage credentials
java.lang.IllegalArgumentException: Invalid credentials endpoint: null
at
org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
at
org.apache.iceberg.aws.s3.VendedCredentialsProvider.<init>(VendedCredentialsProvider.java:61)
at
org.apache.iceberg.aws.s3.VendedCredentialsProvider.create(VendedCredentialsProvider.java:88)
at
org.apache.iceberg.aws.s3.S3FileIO.refreshStorageCredentials(S3FileIO.java:464)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
```
## Root Cause Analysis
`S3FileIO.refreshStorageCredentials()` calls
`VendedCredentialsProvider.create(properties)` passing the raw catalog
properties from the `SerializableMap`.
`VendedCredentialsProvider` expects the property `credentials.uri` (constant
`VendedCredentialsProvider.URI`) to be present in the map. However, the raw
catalog properties only contain:
- `uri` — the catalog base endpoint
- `client.refresh-credentials-endpoint` — the relative path returned by the
REST catalog server (e.g.
`v1/{warehouse}/namespaces/{ns}/tables/{table}/credentials`)
The resolution of `uri` + `client.refresh-credentials-endpoint` →
`credentials.uri` is normally done by `RESTSessionCatalog` when it constructs
the properties for `VendedCredentialsProvider` used by the internal S3 client
(`PrefixedS3Client`). But in the `refreshStorageCredentials()` code path, this
resolution is skipped — the raw properties are passed directly.
### Evidence from bytecode decompilation
```
// VendedCredentialsProvider constructor:
ldc #35 // String "credentials.uri"
invokeinterface Map.get
...
ldc #72 // String "Invalid credentials endpoint: null"
invokestatic Preconditions.checkArgument
```
### Polaris loadTable response (with X-Iceberg-Access-Delegation:
vended-credentials)
```json
{
"s3.access-key-id": "ASIA...",
"s3.secret-access-key": "...",
"s3.session-token": "...",
"client.refresh-credentials-endpoint":
"v1/products_warehouse/namespaces/gold_crd/tables/dim_contract/credentials",
"expiration-time": "1787596581000",
"s3.session-token-expires-at-ms": "1787596581000",
"client.region": "eu-west-1"
}
```
The server correctly provides `client.refresh-credentials-endpoint`, but the
client does not resolve it into `credentials.uri` before passing it to
`VendedCredentialsProvider`.
### Comparison with Iceberg 1.10.0
In 1.10.0, `S3FileIO` does NOT have `refreshStorageCredentials()`. The
`VendedCredentialsProvider` is only used internally by `PrefixedS3Client` where
`credentials.uri` is correctly constructed by the `RESTSessionCatalog`. This is
a regression introduced in 1.11.0 by PR #15678.
## To Reproduce
1. Set up a REST catalog server (e.g., Apache Polaris) with S3 storage and
vended credentials enabled
2. Configure Spark with:
```properties
spark.sql.catalog.mycatalog=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.mycatalog.type=rest
spark.sql.catalog.mycatalog.uri=http://<polaris>/api/catalog
spark.sql.catalog.mycatalog.header.X-Iceberg-Access-Delegation=vended-credentials
spark.sql.catalog.mycatalog.credential=<client_id>:<client_secret>
```
3. Load a table and wait for the credential refresh scheduler to trigger (~5
minutes before credential expiration)
4. Observe the WARN in logs
## Expected behavior
`S3FileIO.refreshStorageCredentials()` should either:
1. Resolve `uri` + `client.refresh-credentials-endpoint` into
`credentials.uri` before calling `VendedCredentialsProvider.create()`
2. Or receive pre-resolved properties from `RESTSessionCatalog`
## Actual behavior
The method passes raw catalog properties to
`VendedCredentialsProvider.create()`, which fails because `credentials.uri` is
absent.
## Current workaround
The warning is non-blocking as long as `cache.expiration-interval-ms`
(default 30s) causes frequent `loadTable` calls that provide fresh credentials
before expiration. No data loss occurs. However, for long-running single writes
(>1h), the credentials could expire without being refreshed.
## Environment
- Iceberg: 1.11.0 (iceberg-spark-runtime-4.0_2.13-1.11.0.jar)
- Spark: 4.0
- REST Catalog: Apache Polaris 1.4.1 (also verified against Polaris 1.7.0
docs — same behavior)
- Storage: AWS S3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
