r0mdau opened a new issue, #805: URL: https://github.com/apache/arrow-rs-object-store/issues/805
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** One of our production process started failing every GCS upload with `403 storage.objects.create denied` and never recovered on its own. Digging in, the trigger was a transient GKE metadata-server fault that handed the pod a perfectly valid token, just for the wrong identity. From that point on every request was rejected, but the 403s did nothing to the cached credential: `object_store` held onto that wrong-identity token for the full ~1h Workload Identity TTL, and the only way out was restarting the process. The reason it can't heal itself is that the `TokenCache` behind the cloud `CredentialProvider`s (AWS / Azure / GCP) is purely expiry-driven, a cached token is reused until it's within `min_ttl` of expiring, and there's no way for a caller to say "this token just got rejected, drop it and fetch a fresh one." So a single bad fetch is sticky: as long as the token hasn't expired, the cache keeps handing it back no matter how many 401/403s the downstream requests return. **Describe the solution you'd like** Add a defaulted, backwards-compatible invalidation hook to the `CredentialProvider` trait. Proof-of-concept against the `v0.13.1` version tag: [r0mdau/arrow-rs-object-store@2961575](https://github.com/r0mdau/arrow-rs-object-store/commit/2961575459044af6f59e7dece22e5408eadba6c0). A caller that observes sustained auth failures can then call `invalidate()`, and the next `get_credential()` re-resolves identity from the underlying source (metadata server, STS, etc.). Providers that don't cache, e.g. `StaticCredentialProvider`, inherit the no-op default and are unaffected. The `clear()` method is feature-gated to `aws` / `azure` / `gcp`, matching the rest of `TokenCache`. **Describe alternatives you've considered** - **Wrap the provider app-side** to interpose our own clearable cache. This duplicates the crate's token logic and can't reach the cache inside the built-in `InstanceCredentialProvider`, which is exactly the one we need to invalidate. - **Wait for expiry.** Up to ~1h of downtime, and it doesn't help when the underlying fault keeps renewing the same bad token. - **A `force_refresh()` variant** instead of clear. `clear()` is simpler and lets the normal fetch path repopulate on the next call; a dedicated refresh method adds surface area for no extra benefit here. **Additional context** Happy to open the PR. The change is purely additive, a new defaulted trait method, one `pub(crate)` cache method, and one provider override. So it should carry no compatibility risk for existing implementors. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
