parthchandra opened a new issue, #6024:
URL: https://github.com/apache/datafusion-comet/issues/6024
### What is the problem the feature request solves?
#### What happens
When Comet's native scan runs on EKS with IRSA (IAM Roles for Service
Accounts), each executor gets
its S3 credentials by calling STS `AssumeRoleWithWebIdentity` to assume the
app role. When many
executors start at once (many pods times many cores), they all make this
call at the same time and
STS throttles it.
The credential chain does not retry the throttled call. Instead it falls
through to the EKS node
instance role, which usually has no access to the data bucket. So every read
then fails with
`403 AccessDenied`, and the job dies — even though the throttle was
temporary and a retry would
have worked.
We saw this on a real run: 238 of 250 executors failed at the same moment,
~2 minutes into the
scan, all with 403s. The exact same query at lower per-pod concurrency
succeeded, because the
credential burst stayed under the STS limit.
#### Why it happens
Three separate problems stack up:
1. **No retry on throttle.** STS reports the throttle as retryable, but the
call is not retried
before the chain gives up.
2. **Silent downgrade.** On failure the chain falls back to the node
instance role instead of
failing loudly. The node role can't read the bucket, so every read 403s.
3. **No shared credential.** Each native reader thread fetches its own
credential, so a pod with N
cores makes N assume-role calls, and they all refresh at the same time.
This is what trips the
STS rate limit.
#### Where
- Native Iceberg scan (`CometIcebergNativeScan`): when no Comet credential
provider is set,
opendal's default credential chain does the assume-role, doesn't retry,
and downgrades.
- Native Parquet scan (`s3.rs`): when no explicit `aws.credentials.provider`
is set, the default
AWS chain puts the node/instance role after web-identity, so the same
downgrade can happen.
### Describe the potential solution
- Retry the throttled assume-role call with backoff and jitter before giving
up.
- Never quietly downgrade to the node role on a transient throttle — surface
a retryable error
instead.
- Fetch one credential per executor process, share it across all reader
threads, and refresh it
early with jitter so refreshes don't line up into another burst.
### Additional context
#### Environment
- Comet native Iceberg scan, Spark 3.4.3, EKS/IRSA, S3-backed Iceberg table.
- Roughly 1000 concurrent native readers (250 executors x 4 cores) at
startup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
