pepijnve commented on issue #25662:
URL: https://github.com/apache/datafusion/issues/25662#issuecomment-5815066386
I queried AI to get an idea of some of the design options we could consider.
Aligns more or less with what I was thinking. The DuckDB approach feels a
little bit too magic to my liking. I think I would prefer something simpler
where you can have a secret value reference in the `CREATE EXTERNAL` options
which is closer to what's described as the RisingWave variant below.
**Prompt**
I’m looking into adding create secret support to DataFusion. A number of
other olap systems also support this with varying syntaxes. Help me weigh the
tradeoffs of the various designs out there.
**Answer**
The designs cluster into a few families, and the choice mostly comes down to
two questions: is the secret a typed bundle of provider config or an opaque
value? And do consumers find it implicitly or reference it by name?
**The main families**
**_Typed secret with implicit scope matching (DuckDB)._**
`CREATE SECRET s (TYPE s3, KEY_ID '...', SECRET '...', REGION '...', SCOPE
's3://bucket/prefix')`. A query touching s3://bucket/prefix/x.parquet picks it
up automatically, with the longest scope prefix winning. It also supports
PROVIDER credential_chain to resolve lazily from the environment, and TEMPORARY
vs PERSISTENT storage.
- Pros: very ergonomic for ad hoc analysis, and users never repeat
credentials in queries.
- Cons: resolution is "magic," so it's harder to audit which secret a
query used. The secret type is also coupled to the consumer (an s3 secret only
means something to the S3 reader). DuckDB's persistent secrets are stored
unencrypted on disk, which is a common criticism.
**_Opaque secret plus a separate connection object (Materialize,
RisingWave)._**
`CREATE SECRET pw AS '...'`, then `CREATE CONNECTION pg TO POSTGRES (...,
PASSWORD SECRET pw)`.
- Pros: clean separation of sensitive and non-sensitive config. You can
rotate the secret without touching the connection, grant privileges on each
separately, and redact easily.
- Cons: more verbose, and it requires a second DDL object (connections)
that DataFusion doesn't have today.
**_Typed secrets with explicit reference (Snowflake)._**
`TYPE = PASSWORD | OAUTH2 | GENERIC_STRING | CLOUD_PROVIDER_TOKEN`, attached
explicitly to integrations or UDFs.
- Pros: types are generic credential shapes rather than per-service, and
usage is explicit and auditable.
- Cons: heavyweight, and tied to Snowflake's RBAC and integration model.
**_Named config collections (ClickHouse `CREATE NAMED COLLECTION`)._**
These are arbitrary key/value bags referenced by name in table functions,
like `s3(my_coll, url='...')`, with per-call overrides.
- Pros: general-purpose.
- Cons: it blurs secret and config. Everything in the bag has to be
treated as sensitive, or nothing is.
**_Governed credential/location objects (Databricks UC storage credentials
plus external locations, BigQuery `CREATE CONNECTION`)._**
Path-based authorization is layered on top of credentials. This is great for
multi-tenant governance but is really a catalog feature, not an engine one.
**_Per-user mappings (Postgres FDW `SERVER` + `USER MAPPING`)._**
This is worth borrowing conceptually if embedders need different credentials
per session or user for the same endpoint.
**DataFusion-specific considerations**
- Parser: sqlparser-rs already parses DuckDB's CREATE SECRET
(Statement::CreateSecret), so the DuckDB syntax is nearly free and familiar to
users.
- Where it lives: DataFusion has no users, auth, or durable catalog of its
own. That argues for a pluggable SecretManager/provider trait on the session or
RuntimeEnv, with in-memory temporary storage as the default and persistence and
encryption left to embedders (Vault, AWS Secrets Manager, and so on).
- Object store fit: ObjectStoreRegistry is URL-keyed, which maps naturally
onto DuckDB-style scopes. However, it resolves at scheme+host (bucket)
granularity, while DuckDB matches arbitrary prefixes. Decide whether
prefix-level scoping is worth changing registry resolution.
- Redaction is the real work: today, CREATE EXTERNAL TABLE ... OPTIONS
('aws.secret_access_key' ...) puts credentials in SQL text, logical plans,
EXPLAIN, Debug output, and potentially proto/Substrait serialization. Whatever
syntax you pick, plans should carry a reference to the secret, never its value,
and resolution should happen as late as possible, ideally at object store
construction.
- Distributed embedders (Ballista, Comet, custom engines) need references
that serialize across the wire and resolve on executors. That favors named
references over values baked into plans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
