vigneshtdev opened a new issue, #116:
URL: https://github.com/apache/datafusion-java/issues/116
**Is your feature request related to a problem or challenge?**
`ObjectStoreOptions.S3` currently supports two credential modes: static
values (`accessKeyId` / `secretAccessKey` / `sessionToken`) passed once at
registration, or falling through to `object_store`'s built-in provider chain
(IMDS instance profile, ECS task endpoint, EKS pod identity, web identity).
For a long-lived `SessionContext` — the intended usage pattern for
DataFusion — this leaves no working credential story in several common
deployments:
- **STS-assumed roles**: IMDS only serves the instance-profile role.
Credentials for an assumed role (cross-account, scoped-down, role chaining) can
only be passed statically, and expire after 1–12h with no refresh. Queries then
start failing with `ExpiredToken` until the store is somehow rebuilt.
- **AWS Lambda**: there is no IMDS endpoint. Credentials arrive via
environment variables, which `object_store` captures once as a
`StaticCredentialProvider`. A `SessionContext` cached across warm invocations
goes stale when the environment credentials rotate underneath it.
- **SSO / profiles / `credential_process` / custom credential vendors**:
not in `object_store`'s chain at all.
Upstream has explicitly declined to widen the built-in chain:
apache/arrow-rs#5143 (use the official AWS SDK) and
apache/arrow-rs-object-store#47 (`credential_process` support) were both closed
as not planned. The sanctioned extension point is
[`AmazonS3Builder::with_credentials(AwsCredentialProvider)`](https://docs.rs/object_store/latest/object_store/aws/struct.AmazonS3Builder.html#method.with_credentials).
Quoting the object_store maintainer in those threads:
> "Users can provide their own credential providers […] This could even be
using the AWS SDK if they so wish."
> "Encouraging the downstreams to expose, or otherwise utilize the
object_store credential provider API would avoid this entirely."
> "I'd naturally prefer the solution that gives users the most flexibility
and avoids needing to revisit this again when someone comes along requesting
SSO or similar."
The Java binding does not expose this extension point, so Java users are
locked out of everything the AWS SDK for Java's default credential chain
already handles. There is direct precedent for fixing this at the binding
layer: arrow-rs-object-store#47 was closed *because* polars exposed a
credential provider
callback in its Python API ("Polars exposed it as part of their API and
configured it to use boto when available. I'm going to close this…").
### Describe the solution you'd like
An S3-specific credentials provider callback on the S3 options builder,
bridged to `AmazonS3Builder::with_credentials`:
```java
/**
* Supplies current AWS credentials. Invoked by the native layer when its
cached
* credentials approach expiry; implementations must be thread-safe.
*/
@FunctionalInterface
public interface AwsCredentialsProvider {
AwsCredentials get();
}
public record AwsCredentials(
String accessKeyId,
String secretAccessKey,
Optional<String> sessionToken,
Optional<Instant> expiration) {}
SessionContext.builder()
.registerObjectStore(
ObjectStoreOptions.s3()
.bucket("my-bucket")
.credentialsProvider(provider) // new
.build())
.build();
// Typical usage delegates to the AWS SDK for Java, which brings the full
default chain (SSO, profiles, credential_process, STS auto-refresh, Lambda env)
for free:
var sdk =
software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.create();
var options = ObjectStoreOptions.s3()
.bucket("my-bucket")
.credentialsProvider(() ->
AwsCredentials.from(sdk.resolveCredentials()))
.build();
```
Native side. A Rust struct holding a JNI GlobalRef to the Java provider
implements object_store::CredentialProvider<Credential = AwsCredential> and is
passed to AmazonS3Builder::with_credentials in build_s3
(native/src/object_store.rs). The bridge caches the returned credentials keyed
on expiration (no expiration
= cache indefinitely) with a min-TTL refresh-ahead check, so the JVM
upcall fires roughly once per credential rotation, not per request. The repo
already has the required upcall machinery: native/src/udf.rs holds a GlobalRef
plus cached bridge class and calls attach_current_thread() to invoke Java from
DataFusion
execution threads — the credentials bridge follows the same pattern.
(object_store's own TokenCache is pub(crate), so the bridge implements its own
small equivalent, as delta-rs does.)
Since registration currently crosses JNI as a serialized protobuf
ObjectStoreRegistration, the provider jobject would be passed alongside the
message rather than inside it — the same way UDF registration passes the UDF
object.
Semantics.
- The provider is called on native (tokio worker) threads; implementations
must be thread-safe.
- Returning an expiration opts into refresh: the native layer re-invokes
the provider before expiry. Because every request resolves credentials at sign
time, rotation is safe mid-query — long scans and multipart uploads spanning a
rotation just sign later requests with the newer credentials.
- Mutually exclusive with static accessKeyId/secretAccessKey on the same
builder — setting both is a build-time error.
- Java exceptions from the provider are translated into
object_store::Error::Generic and surface as query errors.
- The GlobalRef is released when the registered store is dropped.
Non-goals.
- No generic cross-backend credential abstraction: with_credentials is
typed per backend (AwsCredential vs GcpCredential bearer tokens), so this API
belongs on the S3 builder only.
- GCS is out of scope: service-account keys and the metadata server
already self-refresh in object_store. A gcs().credentialsProvider(...)
(bearer-token supplier) can follow the same pattern later if a concrete need
appears.
- No replication of the AWS credential chain inside the binding —
resolving credentials is the AWS SDK for Java's job; the binding only ferries
the result. The binding takes no dependency on the AWS SDK; a convenience
adapter could be a follow-up.
### Describe alternatives you've considered
1. **Do nothing / document the built-in chain.** Leaves STS-assumed roles,
Lambda with a cached context, SSO, and `credential_process` broken; upstream
has frozen the built-in chain (apache/arrow-rs#5143,
apache/arrow-rs-object-store#47), so this gap is permanent.
2. **Imperative `refreshCredentials(newCreds)` method.** Requires the
application to run its own expiry timers, is not atomic with respect to
in-flight queries without extra locking in the binding, and puts lifecycle
behavior on an immutable options object. The pull-based provider + cache is
race-free by
construction (each request resolves current credentials) and matches the
design every AWS SDK converged on.
3. **Re-register the object store with new options.** Same timer problem,
plus undefined behavior for scans already planned against the previously
registered store.
### Additional context
- Maintainer guidance to downstream bindings: apache/arrow-rs#5143
(comments from 2024-08-29 onward) and apache/arrow-rs-object-store#47.
- Precedent: polars exposed exactly this callback in its Python API, which
is what closed arrow-rs-object-store#47 (pola-rs/polars#18979).
- Rust-side reference implementation gluing the AWS SDK to a
`CredentialProvider`: [delta-rs
`credentials.rs`](https://github.com/delta-io/delta-rs/blob/main/crates/aws/src/credentials.rs).
- Existing Rust→Java upcall pattern in this repo: `native/src/udf.rs`
(`GlobalRef` + `attach_current_thread`).
I'm happy to implement this if the approach is agreed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
