sunchao commented on PR #5365: URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5399703778
### [P2] Use Hadoop's long-before-short bucket credential precedence The long-only and divergent-global cases from the [previous comment](https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5389049677) are fixed at `6d20ad69`, but conflicting **long and short bucket keys** still pass the new guard. The [comparison](https://github.com/apache/datafusion-comet/blob/6d20ad69fd39f52c35230d2fa5373aa215e7ed5e/contrib/delta-spark/src/main/scala/org/apache/comet/contrib/delta/DeltaScanSupport.scala#L670-L674) computes Hadoop's effective value as `short.orElse(long).orElse(global)`. Actual Hadoop 3.4.1 resolves the long bucket form first; a nonempty long value is retained rather than overridden by the short form. For example, with distinct fixture credentials: ```text fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider fs.s3a.bucket.mybucket.fs.s3a.access.key=long-ak fs.s3a.bucket.mybucket.fs.s3a.secret.key=long-sk fs.s3a.bucket.mybucket.access.key=short-ak fs.s3a.bucket.mybucket.secret.key=short-sk ``` The current admission helper returns `None`. Hadoop's `SimpleAWSCredentialsProvider` selects the long pair, while native selects the short pair. A Delta read that works through Hadoop can therefore fail authentication or use a different identity after being claimed. I verified this using actual Hadoop 3.4.1 and the current admission/config-extraction methods, both before and after `S3AUtils.propagateBucketOptions`, plus exact-source Rust lookup tests. No live S3 request or full JNI scan was used. Could the plain-value comparison use Hadoop's actual long-before-short precedence, and could the divergent-short regression check the real provider instead of asserting that the short form wins? This is a remaining case of the existing S3-authentication P2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
