dwsmith1983 commented on issue #6575: URL: https://github.com/apache/datafusion-comet/issues/6575#issuecomment-5975831644
Proposal once #5872 lands: route the two SDK names through `HadoopS3ACredentialProviderAdapter` the same way #5872 routes Hadoop's `ProfileAWSCredentialsProvider`. Hadoop maps both names to the Java SDK's `ProfileCredentialsProvider`, so they would resolve exactly as in Spark, including the cases above, and on Spark 3.x through the adapter's SDK v1 build. The change itself is small: the provider class lookup, its table test, and the docs. The trade-off is that these names work today, unlike the profile provider #5872 handles, so routing them adds the adapter's requirements to configurations that run now: - hadoop-aws and the AWS SDK must be visible to Comet's class loader; with Comet on `extraClassPath` and hadoop-aws only through `--packages`, the scan would fail with `NoClassDefFoundError` where it works today; - a profile with static keys asks the provider through JNI on each request. The cases that differ are uncommon profile shapes (keys plus `credential_process` in one source profile, multi-role chains with different regions, a role naming itself as `source_profile`, SSO keys with `role_arn`); a plain profile or a single role mostly works today. I would not fall back to the native path when the adapter cannot load, since the signing identity would then depend on the classpath. @sunchao @andygrove does the classpath requirement for existing configurations look acceptable? If so I'll open the PR after #5872 merges. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
