Hello,
I would like to start this ML for discussing if we should implement the
pagination support for policy APIs (GET policy only for this ML). Here is a
reference report issue for this matter:
https://github.com/apache/polaris/issues/5311.
Currently we have following paths which should support pagination:
1. GET /polaris/v1/{prefix}/namespaces/{namespace}/policies
2. GET /polaris/v1/{prefix}/applicable-policies
For GET policies, we are returning everything via PageToken.readEverything()
(https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/catalog/policy/PolicyCatalog.java#L177C15-L177C41)
For GET applicable-policies, we are computing effective policy by walking the
entity hierarchy in memory rather than issuing an single backing-store listing,
we may need some discussion around this. But it may be worth to start a
different ML for this unless preferred by the community.
Now to honor the pagination for GET policy, there are 2 cases which we will
need to support:
case 1: No filter on policyType
This is the simple case where we just need to thread the token and matching to
listSemanticModels (ref:
https://github.com/apache/polaris/blob/48126190377cfb2d0a54b4b9905a7375fde457ec/extensions/semantic-models/src/main/java/org/apache/polaris/service/catalog/semanticmodel/SemanticModelCatalogAdapter.java#L92).
case 2: Filter on policyType
When a policyType is been set, the current code is calling listFullEntitiesAll
and perform filter in memory (ref:
https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/catalog/policy/PolicyCatalog.java#L188).
For case 2, I would like to see what community thinks regarding what we should
perform:
1. Ignore the pagination when a filter is set?
- In this case, we stay with current behavior
2. Pagination at the post-filter?
- We do pagination on the filtered response
3. Add a filter-aware store mthod?
- Something likes listEntitiesByPolicyType that we handle pagination and avoid
current full fetch and in-memory filter
Thanks,
Yong
