I’d separate two discussions. Adding pagination to the list is a no-brainer for me. We can change the get applicable policies when use cares call.
Yufei On Sun, Sep 13, 2026 at 05:00 Prithvi S <[email protected]> wrote: > Hi Yong, > > Thanks for starting this, and for keeping GET applicable-policies out of > this thread :) I agree listPolicies is the right first cut. > I’d honor pagination in both cases. The spec already advertises page-token > / page-size regardless of policyType, and if we only paginate when the > filter is absent the same endpoint has two contracts. > > Case 1 looks like a straight follow of listSemanticModels (thread PageToken > into listEntities, LinkedHashSet so order is stable). > > For case 2 I would not ignore pagination, and I would not start with a new > store method. policyType lives in entity properties, so > listEntitiesByPolicyType is an SPI + every backend. Post-filter pagination > is enough to honor the REST contract: a page should contain up to pageSize > matching policies, not a store page that is then filtered down. > BasePersistence.listFullEntities already takes an entityFilter and pages > after it. A dedicated store filter can come later if policy volume needs > it. > > wdyt? > > Thanks, > Prithvi S > > On Sun, Sep 13, 2026 at 6:45 AM Yong Zheng <[email protected]> wrote: > > > Hello, > > > > I would like to start this ML for discussing if we should implement the > > pagination support for policy APIs (GET policy only for this ML). Here > is a > > reference report issue for this matter: > > https://github.com/apache/polaris/issues/5311. > > > > Currently we have following paths which should support pagination: > > > > 1. GET /polaris/v1/{prefix}/namespaces/{namespace}/policies > > 2. GET /polaris/v1/{prefix}/applicable-policies > > > > For GET policies, we are returning everything via > > PageToken.readEverything() ( > > > https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/catalog/policy/PolicyCatalog.java#L177C15-L177C41 > > ) > > > > For GET applicable-policies, we are computing effective policy by walking > > the entity hierarchy in memory rather than issuing an single > backing-store > > listing, we may need some discussion around this. But it may be worth to > > start a different ML for this unless preferred by the community. > > > > Now to honor the pagination for GET policy, there are 2 cases which we > > will need to support: > > > > case 1: No filter on policyType > > > > This is the simple case where we just need to thread the token and > > matching to listSemanticModels (ref: > > > https://github.com/apache/polaris/blob/48126190377cfb2d0a54b4b9905a7375fde457ec/extensions/semantic-models/src/main/java/org/apache/polaris/service/catalog/semanticmodel/SemanticModelCatalogAdapter.java#L92 > > ). > > > > case 2: Filter on policyType > > > > When a policyType is been set, the current code is calling > > listFullEntitiesAll and perform filter in memory (ref: > > > https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/catalog/policy/PolicyCatalog.java#L188 > > ). > > > > For case 2, I would like to see what community thinks regarding what we > > should perform: > > 1. Ignore the pagination when a filter is set? > > - In this case, we stay with current behavior > > 2. Pagination at the post-filter? > > - We do pagination on the filtered response > > 3. Add a filter-aware store mthod? > > - Something likes listEntitiesByPolicyType that we handle pagination and > > avoid current full fetch and in-memory filter > > > > Thanks, > > Yong > > >
