j1wonpark commented on issue #7230:
URL: https://github.com/apache/kyuubi/issues/7230#issuecomment-4943622138
Hit the same class of problem with the ClickHouse Spark connector (its
native `TableProvider` landed in 0.10.0).
@pan3793 re: whether `table.properties` is a contract — it isn't.
`Table#properties()` is a `default` method returning `Collections.emptyMap()`,
and the reserved keys `TableCatalog` defines (`location`, `provider`, `owner`,
...) carry metadata, not identity; identity lives in the separate `Identifier`
type. Reading connector-specific property keys means one extraction rule per
connector — tried for Paimon in #5592, closed unmerged once Paimon exposed a
proper identifier.
Where the empty identifier comes from: only `SupportsCatalogOptions`
providers get a catalog/identifier pair. On the read side
`DataSourceV2Utils.loadV2Source` falls back to
```scala
case _ =>
// TODO: Non-catalog paths for DSV2 are currently not well defined.
...
(tbl, None, None, None)
```
and `DataFrameWriter.saveInternal` (`saveCommand` since Spark 4.1) does the
same on the write side — the `[update]` error in this issue is that write path.
So only DSv2 `TableProvider`s without `SupportsCatalogOptions`, used via
`format(...).load()` / `.save()`, are affected: MongoDB, ClickHouse-native,
etc. Batch file sources and Kafka are shielded by the default
`spark.sql.sources.useV1SourceList`, and the built-in `jdbc` format is v1-only.
Two observations about the current state:
1. **When `name()` is a single token — the `MongoTable()` in this issue —
the deny cannot be lifted by any policy.** `PrivilegeObject.dbname` is `null`,
`RangerAccessResourceImpl.setValue(name, null)` drops the key, and Ranger's
`RangerDefaultPolicyResourceMatcher.isHierarchyValidForResources` rejects any
resource that has `table` without its parent `database`: `getMatchType` returns
`NONE` for every policy, including `database=*` / `table=*` (verified against
Ranger 2.6.0, the version Authz builds with). An unmatched request is denied by
default, so this isn't enforcement — it's a deny no policy can express an allow
for. (The `FUNCTION` branch of `AccessResource` already coerces a null db to
`""`, which avoids exactly this.) Multi-token names fare no better:
ClickHouse's `` `db`.`table` `` splits into backquoted tokens that never match
the operator's real `db`/`table` policies — only wildcards do. Either way the
plugin is authorizing fabricated resource names.
2. **The v1 equivalent is already skipped, unconditionally.** A
`LogicalRelation` without `catalogTable` (e.g.
`spark.read.format("jdbc").load()`) yields no privilege object at all. v1
direct reads pass silently; v2 direct reads hard-fail.
As @bowenliang123 put it in #5592, "Authz requires proper identifier from
logical plan for privilege checks" — and these relations structurally cannot
provide one.
Proposal: an operator-level escape hatch, default behavior unchanged —
`spark.kyuubi.authz.skip.catalogless.v2.relation.enabled`, default `false`.
When enabled, the extractor returns `None` when catalog and identifier are both
empty, matching the v1 path. Whether skipping is safe is deployment-specific:
fine when credentials for the external system are supplied per query by the end
user, risky when the shared Spark principal holds ambient credentials — hence
opt-in rather than unconditional. The key should be barred from in-session
`SET` like `spark.sql.runSQLOnFiles`, recommended for
`kyuubi.session.conf.restrict.list`, and documented as producing no Ranger
audit events for skipped relations.
(Considered and deferred: parsing multi-token `name()`s into db/table — the
Ranger resource has no catalog dimension, so identifiers from user-chosen
external endpoints would share the namespace of operator-configured catalogs.
Worth its own discussion.)
I've opened #7557 implementing the conf-gated version — if the preference is
rather unconditional skip for exact v1 parity, I'm happy to simplify the PR
accordingly. WDYT?
@moelhoussein feel free to take over if you're still working on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
