ametel01 opened a new pull request, #23325: URL: https://github.com/apache/datafusion/pull/23325
## Which issue does this PR close? - Closes #23321. ## Rationale for this change The CLI's S3 credential provider wrapped AWS SDK credentials but called `SharedCredentialsProvider::provide_credentials()` for every object-store credential request. That bypasses the AWS SDK request pipeline identity cache and can repeatedly invoke expensive providers such as `credential_process` or SSO-backed providers even when the returned credentials have not expired. ## What changes are included in this PR? - Cache the converted `object_store::aws::AwsCredential` inside `S3CredentialProvider`. - Seed that cache from the initial credential fetch already performed while building the S3 object store. - Reuse cached credentials until they are expired or within a small refresh buffer. - Add regression coverage for seeded credentials, non-expiring credentials, expired and near-expiry credentials, and an end-to-end `credential_process` flow that fails on `main` without this fix. ## Are these changes tested? Yes: - `cargo test -p datafusion-cli s3_object_store_reuses_fetched_credentials_until_expiry -- --nocapture` - `cargo test -p datafusion-cli object_storage` - `cargo test -p datafusion-cli` - `cargo clippy --all-targets --all-features -- -D warnings` I also verified the new public-flow regression test against detached `main`; it fails there because the credential process runs multiple times instead of once. ## Are there any user-facing changes? No API or configuration changes. The CLI should make fewer redundant AWS credential-provider calls for S3 object stores while still refreshing expiring credentials. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
