yuqi1129 opened a new issue, #12622:
URL: https://github.com/apache/gravitino/issues/12622
### What would you like to be improved?
`GET /metalakes/{metalake}/catalogs?details=true` still runs one
authorization evaluation per catalog, even when the caller holds a
metalake-scope grant that makes every catalog visible. The short-circuit added
in #11775 does not reach it.
`CatalogOperations.listCatalogs` calls two different overloads of
`MetadataAuthzHelper.filterByExpression`:
| Branch | Passes | Overload
| Short-circuit | Preload |
|-----------------|------------------------|---------------------------------------------|---------------|---------|
| `details=false` | `NameIdentifier[]` | `filterByExpression(...,
NameIdentifier[])` | yes | yes |
| `details=true` | `Catalog[]` + a mapper | `filterByExpression(..., E[],
Function)` | no | no |
The second overload goes straight to `doFilter`, so
`allVisibleViaParentScope` is never consulted and `preloadToCache` /
`preloadOwner` never run. #11775 described "one insertion point covers
listTables/listSchemas/listCatalogs", which holds for tables and schemas
because their list endpoints only return identifiers. Catalogs are the one
entity with a `details=true` variant, so they were left on the per-object path.
Measured on a metalake with about 1300 catalogs, authorization enabled,
where every observed call used `details=true`:
- p50 7.3s, p90 113s, max 126s
- 634 requests over three minutes, 150 of which never produced a response
- peak concurrent list requests per node above 100
The authorization filter submits one task per object to a shared fixed pool
(`gravitino.authorization.threadPoolSize`, default 100), so roughly a hundred
concurrent requests each queue about 1300 tasks into the same pool.
The same overload is used by the list endpoints for functions, groups, jobs,
policies, roles, tags and users, so none of those get the preloads either.
### How should we improve?
Move the short-circuit and the two preloads into the generic overload and
let the `NameIdentifier[]` overload delegate to it, so every list shape passes
the same point and the two cannot drift apart again.
A second, independent cost on this endpoint is that
`CatalogManager.listCatalogsInfo` resolves properties for every catalog through
`loadCatalogAndWrap` before the filter runs, so invisible catalogs are still
instantiated. That is worth handling separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
