mbutrovich commented on issue #2341: URL: https://github.com/apache/datafusion-comet/issues/2341#issuecomment-3633089618
> So our current blocker is that OpenDAL/Requisiing doesn't support passing custom credential providers written in Java. And once we (or Comet) build such a bridge, we can switch to using OpenDAL? I think if we can come up with an API that makes sense for arbitrary authentication providers to be passed in like the `EncryptionFactory` in DataFusion... https://github.com/apache/datafusion/blob/20870c18a418ec081d44ecf8a90a30a95aa53138/datafusion-examples/examples/data_io/parquet_encrypted_with_kms.rs#L217 ...then we might be in business. Basically, Parquet Modular Encryption allows for arbitrary key provider implementations, so you can define a class that unwraps keys with whatever code it sees fit. In Comet, we use that API to build a JNI bridge to any Java/Scala PME key providers a Spark user might have. https://github.com/apache/datafusion-comet/pull/2447/files I'm imagining a similar design for authentication, but I'm not sure it just handles signing requests or works at a different abstraction. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
