Hi Yong, That's a good call (I think I mentioned that we should start a discussion in a PR I reviewed).
BasePersistence.listFullEntities already takes an entityFilter + PageToken, and AtomicOperationMetaStoreManager already uses that exact pattern elsewhere (the task listing , catalog_role, etc). So this is a valid path. The missing part is that PolarisMetaStoreManager.listFullEntities hardcodes entity, so PolicyCatalog can't reach the filter-aware version. I think we have to add a small filter-aware overload at the manager layer to thread a policyType predicate down. That's a small approach we can implement easily. Regards JB On Sun, Sep 13, 2026 at 3:14 AM Yong Zheng <[email protected]> wrote: > > Hello, > > I would like to start this ML for discussing if we should implement the > pagination support for policy APIs (GET policy only for this ML). Here is a > reference report issue for this matter: > https://github.com/apache/polaris/issues/5311. > > Currently we have following paths which should support pagination: > > 1. GET /polaris/v1/{prefix}/namespaces/{namespace}/policies > 2. GET /polaris/v1/{prefix}/applicable-policies > > For GET policies, we are returning everything via PageToken.readEverything() > (https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/catalog/policy/PolicyCatalog.java#L177C15-L177C41) > > For GET applicable-policies, we are computing effective policy by walking the > entity hierarchy in memory rather than issuing an single backing-store > listing, we may need some discussion around this. But it may be worth to > start a different ML for this unless preferred by the community. > > Now to honor the pagination for GET policy, there are 2 cases which we will > need to support: > > case 1: No filter on policyType > > This is the simple case where we just need to thread the token and matching > to listSemanticModels (ref: > https://github.com/apache/polaris/blob/48126190377cfb2d0a54b4b9905a7375fde457ec/extensions/semantic-models/src/main/java/org/apache/polaris/service/catalog/semanticmodel/SemanticModelCatalogAdapter.java#L92). > > case 2: Filter on policyType > > When a policyType is been set, the current code is calling > listFullEntitiesAll and perform filter in memory (ref: > https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/catalog/policy/PolicyCatalog.java#L188). > > For case 2, I would like to see what community thinks regarding what we > should perform: > 1. Ignore the pagination when a filter is set? > - In this case, we stay with current behavior > 2. Pagination at the post-filter? > - We do pagination on the filtered response > 3. Add a filter-aware store mthod? > - Something likes listEntitiesByPolicyType that we handle pagination and > avoid current full fetch and in-memory filter > > Thanks, > Yong
