parthchandra opened a new issue, #6022: URL: https://github.com/apache/datafusion-comet/issues/6022
### What is the problem the feature request solves? When Comet's native Parquet reader is enabled, an S3 query fails if `fs.s3a.aws.credentials.provider` names a provider class that Comet doesn't recognize, even though plain Spark runs the same query fine. The error looks like: ``` org.apache.comet.CometNativeException: General execution error with reason: Generic S3 error: Unsupported credential provider: com.amazonaws.auth.DefaultAWSCredentialsProviderChain ``` Seen on Spark 3.4.3 / Hadoop 3.3.4. The task retries and the stage aborts. #### Why it happens Comet's native readers don't go through Hadoop S3A. They resolve S3 credentials in Rust and only understand a fixed list of provider class names. Any class not on that list is rejected. Plenty of valid, widely-used classes aren't on it — the AWS default chain, the web-identity wrapper, custom providers — so config that works in Spark breaks under Comet. Iceberg reads don't hit this because that path falls back to a default credential chain instead of failing. So on the same cluster, Iceberg reads succeed while Parquet reads fail. #### Impact - Clusters using a provider class outside Comet's list can't use the native Parquet scan. - Behavior differs from Spark, and differs between Comet's own Parquet and Iceberg paths. ### Describe the potential solution Comet already has an SPI (`CometS3CredentialProvider`) for calling JVM credential code from native, but ships no ready-made implementations. Add two built-in ones so standard providers work with a one-line config change, and forward the `fs.s3a.*` config to the native side so those implementations can read it. This is for the Parquet path only, and is opt-in. ### Additional context - Spark: 3.4.3 - Hadoop: 3.3.4 (AWS SDK v1) - `fs.s3a.aws.credentials.provider = com.amazonaws.auth.DefaultAWSCredentialsProviderChain` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
