jarosmpost opened a new pull request, #17789:
URL: https://github.com/apache/iceberg/pull/17789
## Problem
When Iceberg reads LakeFormation-governed tables at high executor counts
(e.g. 50× G.2X Spark workers), `LakeFormationCredentialsProvider` calls
`GetTemporaryGlueTableCredentials` once per `resolveCredentials()` invocation.
Because this is called on every S3 file open, hundreds of concurrent tasks
simultaneously hit the LakeFormation API, causing:
LakeFormationException: Rate exceeded
This was observed in production: a job that succeeded at 25× G.1X (12,281
API calls) started failing after scaling to 50× G.2X. Reducing executors back
below the rate limit was the only workaround.
## Solution
Wrap the `GetTemporaryGlueTableCredentials` call in a
`CachedSupplier<AwsCredentials>` (the same AWS SDK mechanism already used by
`VendedCredentialsProvider`) so each executor reuses a credential for its full
TTL window instead of fetching on every file open.
The `staleTime` is set to `expiration - refreshLeadTimeMs` (default 60 s),
ensuring credentials are refreshed before they expire.
Two new `AwsProperties` constants control the behaviour:
| Property | Default | Description |
|---|---|---|
| `lakeformation.cache.enabled` | `true` | Set to `false` to disable caching
(e.g. for debugging) |
| `lakeformation.cache.refresh-lead-time-ms` | `60000` | Ms before expiry at
which the cache is proactively refreshed |
Caching is **on by default** — no config change required for existing users.
The fix is contained entirely within the existing
`LakeFormationCredentialsProvider` inner class and
`LakeFormationAwsClientFactory`; no new classes are introduced.
## Validation
Validated on a 50× G.2X Spark job against a LakeFormation-governed
cross-account Iceberg table:
| | API calls | Result |
|---|---|---|
| Before fix | 12,281 | `LakeFormationException: Rate exceeded` |
| After fix | 598 | Job succeeded in 3,393 s |
## Changes
- `AwsProperties`: 2 new constants (`LAKE_FORMATION_CACHE_ENABLED`,
`LAKE_FORMATION_CACHE_REFRESH_LEAD_TIME_MS`) with defaults
- `LakeFormationAwsClientFactory`: read new properties in `initialize()`;
pass to `LakeFormationCredentialsProvider` constructor; add `CachedSupplier`
caching path in the inner class
- `TestLakeFormationCredentialsProvider`: new test class (5 tests) covering
cache hit, cache miss after stale time, and cache-disabled path
---
**AI Disclosure**
- Model: claude-sonnet-4.6
- Platform/Tool: OpenCode
- Human Oversight: fully reviewed
- Prompt Summary: Implement credential caching in
LakeFormationAwsClientFactory to prevent LakeFormation rate exceeded errors at
high Spark executor counts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
