laxmanmamididd opened a new issue, #7390: URL: https://github.com/apache/datafusion-comet/issues/7390
### Describe the bug Comet keys provider instances by `(FQCN, dispatchKey, catalogProperties)`. When an Iceberg REST catalog vends new credentials for the same table, the FileIO properties change, so Comet creates a new provider instance under a new key. The old instance stays cached, and `close()` runs only from the JVM shutdown hook. Reported in review of #7363: three credential snapshots against one refresh endpoint and the same dispatch key left three providers retained. Each Iceberg `VendedCredentialsProvider` owns an HTTP client and an auth session, so a long-lived executor that reloads tables (Spark Connect, Thrift Server) keeps accumulating them. ### Expected behavior The dispatcher retires a provider once a newer instance supersedes it for the same `(FQCN, dispatchKey)` and its active readers have finished, then calls `close()`. ### Additional context This affects every provider, not only `IcebergVendedS3CredentialProvider`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
