Hi Prithvi/Yufei, Thanks for the quick review and response. I do agreed they should both be fixed eventually as that is what the spec stated that is supported (but not honored as of today). As we all agreed the GET policies is a low hanging fruit, I would proceed with this change later this weekend.
Regarding GET applicable-policies, i do want to keep it in a different discussion as stated in this thread as this does require a good amount of changes and potentially more discussions among community. i would keep this ML open while we can get more feedbacks from community. Again, thanks for the quick review and response. I will share the PR once ready (next week if not completed this week). Thanks, Yong On 2026/09/13 17:14:18 Yufei Gu wrote: > I’d separate two discussions. Adding pagination to the list is a no-brainer > for me. We can change the get applicable policies when use cares call. > > > Yufei > > On Sun, Sep 13, 2026 at 05:00 Prithvi S <[email protected]> wrote: > > > Hi Yong, > > > > Thanks for starting this, and for keeping GET applicable-policies out of > > this thread :) I agree listPolicies is the right first cut. > > I’d honor pagination in both cases. The spec already advertises page-token > > / page-size regardless of policyType, and if we only paginate when the > > filter is absent the same endpoint has two contracts. > > > > Case 1 looks like a straight follow of listSemanticModels (thread PageToken > > into listEntities, LinkedHashSet so order is stable). > > > > For case 2 I would not ignore pagination, and I would not start with a new > > store method. policyType lives in entity properties, so > > listEntitiesByPolicyType is an SPI + every backend. Post-filter pagination > > is enough to honor the REST contract: a page should contain up to pageSize > > matching policies, not a store page that is then filtered down. > > BasePersistence.listFullEntities already takes an entityFilter and pages > > after it. A dedicated store filter can come later if policy volume needs > > it. > > > > wdyt? > > > > Thanks, > > Prithvi S > > > > On Sun, Sep 13, 2026 at 6:45 AM Yong Zheng <[email protected]> wrote: > > > > > Hello, > > > > > > I would like to start this ML for discussing if we should implement the > > > pagination support for policy APIs (GET policy only for this ML). Here > > is a > > > reference report issue for this matter: > > > https://github.com/apache/polaris/issues/5311. > > > > > > Currently we have following paths which should support pagination: > > > > > > 1. GET /polaris/v1/{prefix}/namespaces/{namespace}/policies > > > 2. GET /polaris/v1/{prefix}/applicable-policies > > > > > > For GET policies, we are returning everything via > > > PageToken.readEverything() ( > > > > > https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/catalog/policy/PolicyCatalog.java#L177C15-L177C41 > > > ) > > > > > > For GET applicable-policies, we are computing effective policy by walking > > > the entity hierarchy in memory rather than issuing an single > > backing-store > > > listing, we may need some discussion around this. But it may be worth to > > > start a different ML for this unless preferred by the community. > > > > > > Now to honor the pagination for GET policy, there are 2 cases which we > > > will need to support: > > > > > > case 1: No filter on policyType > > > > > > This is the simple case where we just need to thread the token and > > > matching to listSemanticModels (ref: > > > > > https://github.com/apache/polaris/blob/48126190377cfb2d0a54b4b9905a7375fde457ec/extensions/semantic-models/src/main/java/org/apache/polaris/service/catalog/semanticmodel/SemanticModelCatalogAdapter.java#L92 > > > ). > > > > > > case 2: Filter on policyType > > > > > > When a policyType is been set, the current code is calling > > > listFullEntitiesAll and perform filter in memory (ref: > > > > > https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/catalog/policy/PolicyCatalog.java#L188 > > > ). > > > > > > For case 2, I would like to see what community thinks regarding what we > > > should perform: > > > 1. Ignore the pagination when a filter is set? > > > - In this case, we stay with current behavior > > > 2. Pagination at the post-filter? > > > - We do pagination on the filtered response > > > 3. Add a filter-aware store mthod? > > > - Something likes listEntitiesByPolicyType that we handle pagination and > > > avoid current full fetch and in-memory filter > > > > > > Thanks, > > > Yong > > > > > >
