vishal210893 opened a new issue, #13596:
URL: https://github.com/apache/iceberg/issues/13596
### Apache Iceberg version
1.9.1
### Query engine
Other
### Please describe the bug š
## Background
Iām trying to use Apache Icebergās `GlueCatalog` against an S3-backed
warehouse with **temporary credentials** (Access Key, Secret Key, **Session
Token**).
* AWS CLI (`aws s3 ls ā¦`) works fine with the exact same env-vars.
* With Iceberg + GlueCatalog, I repeatedly get:
```
S3Exception: The request signature we calculated does not match the
signature you provided...
```
or
```
AccessDeniedException: 403 Forbidden
```
* Iāve tried:
* Switching `s3://` ā `s3a://` and Hadoop/S3A configs
* Using Icebergās `rest.*` override properties
* Writing a full `StaticAwsClientFactory` that implements **all**
`AwsClientFactory` methods and injecting it via `client.factory`
* Forcing `io-impl = org.apache.iceberg.aws.s3.S3FileIO`
* Verifying in debug logs that my factory **is** called
* Inspecting the AWS-SDK HTTP wire logs for the `x-amz-security-token`
header
* Ensuring my system clock is in sync
Nothing stops Iceberg from signing requests incorrectly or losing the
session token headerāevery S3 call is ultimately rejected as 403 or
signature-mismatch.
---
## Environment
* **Iceberg**: 1.9.1
* **iceberg-aws**: 1.9.1
* **iceberg-parquet**: 1.9.1
* **AWS SDK v2 modules**:
* `software.amazon.awssdk:glue`
* `software.amazon.awssdk:sts`
* `software.amazon.awssdk:s3`
* **Java**: 11
* **OS**: macOS / Linux (client in IST time zone)
* **Credentials**: temporary STS credentials with session token
---
## Reproduction Steps
1. **pom.xml dependencies** (as in my project):
```xml
<!-- Iceberg Core -->
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-core</artifactId>
<version>1.9.1</version>
</dependency>
<!-- Iceberg AWS/Glue support -->
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-aws</artifactId>
<version>1.9.1</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>glue</artifactId>
</dependency>
<!-- Iceberg Parquet -->
<dependency>
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-parquet</artifactId>
<version>1.9.1</version>
</dependency>
<!-- AWS SDK v2 for STS and S3 -->
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>sts</artifactId>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
</dependency>
```
2. **GlueCatalogController.java**
I configure both system properties and catalog props, then call
`catalog.loadTable(...)`:
```java
public class GlueCatalogController {
public ResponseEntity<Object> getFilteredData() {
try {
// 1) System props (AWS SDK picks these up)
System.setProperty("aws.accessKeyId", "<ACCESS_KEY_ID>");
System.setProperty("aws.secretAccessKey", "<SECRET_ACCESS_KEY>");
System.setProperty("aws.sessionToken", "<SESSION_TOKEN>");
System.setProperty("aws.region", "us-east-1");
// 2) Iceberg catalog properties
Map<String, String> props = new HashMap<>();
props.put("warehouse", "s3://my-iceberg-bucket/warehouse");
props.put("io-impl", "org.apache.iceberg.aws.s3.S3FileIO");
props.put("catalog-impl","org.apache.iceberg.aws.glue.GlueCatalog");
props.put("glue.id", "<AWS_ACCOUNT_ID>");
props.put("glue.endpoint","https://glue.us-east-1.amazonaws.com";);
// 3) Attempt to wire in my factory
props.put("client.factory",
"com.fdl.controller.StaticAwsClientFactory");
GlueCatalog catalog = new GlueCatalog();
catalog.initialize("glue", props);
Table table = catalog.loadTable(TableIdentifier.of(
"my_db","my_table"
));
log.info("Schema: {}", table.schema().asStruct());
return ResponseEntity.ok().build();
} catch (Exception e) {
log.error("Failed to load Iceberg table", e);
throw e;
}
}
}
```
3. **StaticAwsClientFactory.java**
Full implementation of **all** calls:
```java
public class StaticAwsClientFactory implements AwsClientFactory {
private String accessKeyId, secretAccessKey, sessionToken, region;
// Iceberg will call this
@Override
public void initialize(Map<String, String> props) {
// These exact keys must match AwsClientProperties constants:
accessKeyId =
props.get(AwsClientProperties.CLIENT_FACTORY_ACCESS_KEY_ID);
secretAccessKey =
props.get(AwsClientProperties.CLIENT_FACTORY_SECRET_ACCESS_KEY);
sessionToken =
props.get(AwsClientProperties.CLIENT_FACTORY_SESSION_TOKEN);
region =
props.get(AwsClientProperties.CLIENT_FACTORY_REGION);
System.out.println("Factory init ā token: " +
(sessionToken == null ? "null" : sessionToken.substring(0,10) +
"ā¦"));
}
private StaticCredentialsProvider creds() {
AwsSessionCredentials creds = AwsSessionCredentials.create(
accessKeyId, secretAccessKey, sessionToken);
return StaticCredentialsProvider.create(creds);
}
@Override
public S3Client s3() {
System.out.println("Building S3Client with token: " +
(sessionToken == null ? "null" : sessionToken.substring(0,10) +
"ā¦"));
return S3Client.builder()
.credentialsProvider(creds())
.region(Region.of(region))
.build();
}
@Override
public GlueClient glue() {
return GlueClient.builder()
.credentialsProvider(creds())
.region(Region.of(region))
.build();
}
@Override
public KmsClient kms() {
return KmsClient.builder()
.credentialsProvider(creds())
.region(Region.of(region))
.build();
}
@Override
public DynamoDbClient dynamo() {
return DynamoDbClient.builder()
.credentialsProvider(creds())
.region(Region.of(region))
.build();
}
}
```
4. **What actually happens**
* Debug logs confirm `StaticAwsClientFactory.initialize()` and `.s3()`
**do** run.
* Yet **every** S3 request still returns 403 or signature-mismatch.
* Iāve enabled AWS SDK wire logs and **never** see an
`x-amz-security-token` header, so the session token never makes it to S3.
---
## Expected Behavior
1. `GlueCatalog` should use my `StaticAwsClientFactory` for **all** AWS
calls (Glue, S3, KMS, DynamoDB).
2. Every S3 request should be signed with the provided session token (i.e.
`x-amz-security-token` must appear in the signed headers).
3. `catalog.loadTable(...)` should succeed and return the Iceberg table
metadata.
---
## Actual Behavior
* Iceberg/Hadoop/S3A or Icebergās default FileIO is **silently ignoring**
the session token.
* S3 rejects the request with 403 or signature mismatch.
* No combination of `rest.*` props or
`EnvironmentVariableCredentialsProvider` or client-factory wiring seems to fix
it.
---
## Workarounds Tried
* Switching **only** to `s3a://` + Hadoop/S3A configs (`fs.s3a.*`)
* Using Icebergās **`rest.access-key-id`**, **`rest.secret-access-key`**,
**`rest.session-token`**, **`rest.signer.region`**, **`rest.signing.name`**
properties
* Writing a **custom AwsClientFactory** (full code above)
* Forcing `io-impl = org.apache.iceberg.aws.s3.S3FileIO`
* Remapping `s3://` ā S3A via `fs.s3.impl =
org.apache.hadoop.fs.s3a.S3AFileSystem`
* Ensuring the exact property keys:
* `client.factory` = fully-qualified factory class
* `catalog-impl`, `warehouse`, `glue.id`, `glue.endpoint`, `io-impl`
* Syncing system clock (no skew)
* Verifying AWS CLI works with the same creds
---
## Questions / Next Steps
1. **Is this a bug** in `GlueCatalog` or in Icebergās AWS client-factory
wiring that drops the session token?
2. **Which configuration** or code change am I still missing to get
`x-amz-security-token` included?
3. Should Icebergās docs be updated to show exactly how to supply
**temporary credentials** to a GlueCatalog?
Any guidance or a fix would be greatly appreciatedāIām completely stuck.
Thank you!
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
