Hi Yong, Thanks for starting this, and for keeping GET applicable-policies out of this thread :) I agree listPolicies is the right first cut. I’d honor pagination in both cases. The spec already advertises page-token / page-size regardless of policyType, and if we only paginate when the filter is absent the same endpoint has two contracts.
Case 1 looks like a straight follow of listSemanticModels (thread PageToken into listEntities, LinkedHashSet so order is stable). For case 2 I would not ignore pagination, and I would not start with a new store method. policyType lives in entity properties, so listEntitiesByPolicyType is an SPI + every backend. Post-filter pagination is enough to honor the REST contract: a page should contain up to pageSize matching policies, not a store page that is then filtered down. BasePersistence.listFullEntities already takes an entityFilter and pages after it. A dedicated store filter can come later if policy volume needs it. wdyt? Thanks, Prithvi S On Sun, Sep 13, 2026 at 6:45 AM Yong Zheng <[email protected]> wrote: > Hello, > > I would like to start this ML for discussing if we should implement the > pagination support for policy APIs (GET policy only for this ML). Here is a > reference report issue for this matter: > https://github.com/apache/polaris/issues/5311. > > Currently we have following paths which should support pagination: > > 1. GET /polaris/v1/{prefix}/namespaces/{namespace}/policies > 2. GET /polaris/v1/{prefix}/applicable-policies > > For GET policies, we are returning everything via > PageToken.readEverything() ( > https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/catalog/policy/PolicyCatalog.java#L177C15-L177C41 > ) > > For GET applicable-policies, we are computing effective policy by walking > the entity hierarchy in memory rather than issuing an single backing-store > listing, we may need some discussion around this. But it may be worth to > start a different ML for this unless preferred by the community. > > Now to honor the pagination for GET policy, there are 2 cases which we > will need to support: > > case 1: No filter on policyType > > This is the simple case where we just need to thread the token and > matching to listSemanticModels (ref: > https://github.com/apache/polaris/blob/48126190377cfb2d0a54b4b9905a7375fde457ec/extensions/semantic-models/src/main/java/org/apache/polaris/service/catalog/semanticmodel/SemanticModelCatalogAdapter.java#L92 > ). > > case 2: Filter on policyType > > When a policyType is been set, the current code is calling > listFullEntitiesAll and perform filter in memory (ref: > https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/catalog/policy/PolicyCatalog.java#L188 > ). > > For case 2, I would like to see what community thinks regarding what we > should perform: > 1. Ignore the pagination when a filter is set? > - In this case, we stay with current behavior > 2. Pagination at the post-filter? > - We do pagination on the filtered response > 3. Add a filter-aware store mthod? > - Something likes listEntitiesByPolicyType that we handle pagination and > avoid current full fetch and in-memory filter > > Thanks, > Yong >
