parthchandra commented on PR #1817: URL: https://github.com/apache/datafusion-comet/pull/1817#issuecomment-2928171165
> > I have another thought on this. Any number of users have developed custom `AWSCredentialsProvider`s in Java but we would not have corresponding implementations in Rust (the AWS Rust api is rather new). One way is to have a mechanism by which we can call the configured credentials provider in the Java side and simply pass in the session credentials to the native side instead of the conf. Another option is to invoke a configured credentials provider from the native side via JNI. Do you think this implementation can be extended to support something like this (later, as an enhancement to this PR)? > > I think calling the Java credential provider from the native via JNI is a better option. The credential may need to be refreshed when we scan the parquet files. If we pass the session credential when creating the scanner, the session credential may expire mid-scan and cause authentication failures. > > The credential provider APIs in both AWS SDK V1 and V2 are relatively straightforward: > > * **V1**: `AWSCredentialsProvider.getCredentials()` → `AWSCredentials` > * **V2**: `AwsCredentialsProvider.resolveCredentials()` → `AwsCredentials` > > So it won't be hard to call it from native side using JNI. One nuance is that AWS SDK V1 and V2 defines credential provider and credential classes differently, our JNI-based native credential provider have to support both versions. I agree, I thought of the same thing some time after I wrote the above comment. I agree, the credentials provider api is pretty straightforward and the JNI based provider can wrap both the V1 and V2 providers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
