sunchao commented on PR #5365: URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5389049677
### [P2] Also handle plain long-form S3 bucket credentials The long-form JCEKS cases are fixed, but the same compatibility gap remains for **plain configuration values** at `514f71e0`. With `fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider` and only these plain keys configured: ```text fs.s3a.bucket.mybucket.fs.s3a.access.key fs.s3a.bucket.mybucket.fs.s3a.secret.key ``` [`bucketCredentialAliasReason`](https://github.com/apache/datafusion-comet/blob/514f71e0d09f4b0e85964014afae17363fa2dba1/contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala#L712-L713) returns `None` because no credential-provider path is set. Hadoop resolves these keys, but `NativeConfig` forwards them unchanged and the [native lookup](https://github.com/apache/datafusion-comet/blob/514f71e0d09f4b0e85964014afae17363fa2dba1/native/core/src/parquet/objectstore/s3.rs#L299-L318) reads only short bucket keys and globals. The native Simple provider therefore has no credentials. If different plain global credentials are also configured, Hadoop uses the long bucket aliases while native uses those globals instead. Claiming the Delta scan exposes this mismatch to reads that previously used Hadoop successfully. I verified both cases with actual Hadoop 3.4.1 `SimpleAWSCredentialsProvider` and the unchanged current admission/extraction methods, before and after `S3AUtils.propagateBucketOptions`, plus exact-source Rust lookup tests. Short-only and global-only controls agree. This was not a full Comet/JNI scan or a live S3 request. Could we conservatively decline these configurations, or normalize the effective credentials using Hadoop's precedence, and add plain-long-only and global-shadowing regressions? This is a remaining case of the [existing S3-authentication P2](https://github.com/apache/datafusion-comet/pull/5365#discussion_r3835076674), posted here for visibility rather than as another inline thread. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
