laxmanmamididd opened a new issue, #7387: URL: https://github.com/apache/datafusion-comet/issues/7387
### Describe the bug With an Iceberg REST catalog that vends short-lived S3 credentials, the native Iceberg scan (`spark.comet.scan.icebergNative.enabled=true`) signs every S3 request with the `s3.*` credentials vended when the driver loaded the table. Comet never refreshes them. A scan that is still reading after they expire fails with `ExpiredToken`. Spark's own read of the same table keeps working, because Iceberg's `S3FileIO` refreshes the credentials through the `client.refresh-credentials-endpoint` the catalog returns. Nothing in the error points at `spark.sql.catalog.<catalog>.s3.comet.credential.provider.class`, which is the only way to avoid this today. ### Steps to reproduce 1. Configure an Iceberg REST catalog that vends S3 credentials with a short lifetime and returns `client.refresh-credentials-endpoint` (for example Unity Catalog, about one hour). 2. Enable Comet with the native Iceberg scan. 3. Run a query whose scan reads for longer than the credential lifetime. ### Expected behavior The native scan refreshes the vended credentials the way `S3FileIO` does, with no extra configuration. ### Additional context Spark 4.0.1, Comet 1.1.0. #7363 adds a provider that refreshes and selects it automatically when the catalog returns a refresh endpoint. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
