Hi Yong, +1 to adding pagination to GET policies.
Looking forward to implementation PR(s). Cheers, Dmitri. On Sun, Sep 13, 2026 at 11:36 PM Yong Zheng <[email protected]> wrote: > Hi Prithvi/Yufei, > > Thanks for the quick review and response. I do agreed they should both be > fixed eventually as that is what the spec stated that is supported (but not > honored as of today). As we all agreed the GET policies is a low hanging > fruit, I would proceed with this change later this weekend. > > Regarding GET applicable-policies, i do want to keep it in a different > discussion as stated in this thread as this does require a good amount of > changes and potentially more discussions among community. i would keep this > ML open while we can get more feedbacks from community. > > Again, thanks for the quick review and response. I will share the PR once > ready (next week if not completed this week). > > Thanks, > Yong > > On 2026/09/13 17:14:18 Yufei Gu wrote: > > I’d separate two discussions. Adding pagination to the list is a > no-brainer > > for me. We can change the get applicable policies when use cares call. > > > > > > Yufei > > > > On Sun, Sep 13, 2026 at 05:00 Prithvi S <[email protected]> > wrote: > > > > > Hi Yong, > > > > > > Thanks for starting this, and for keeping GET applicable-policies out > of > > > this thread :) I agree listPolicies is the right first cut. > > > I’d honor pagination in both cases. The spec already advertises > page-token > > > / page-size regardless of policyType, and if we only paginate when the > > > filter is absent the same endpoint has two contracts. > > > > > > Case 1 looks like a straight follow of listSemanticModels (thread > PageToken > > > into listEntities, LinkedHashSet so order is stable). > > > > > > For case 2 I would not ignore pagination, and I would not start with a > new > > > store method. policyType lives in entity properties, so > > > listEntitiesByPolicyType is an SPI + every backend. Post-filter > pagination > > > is enough to honor the REST contract: a page should contain up to > pageSize > > > matching policies, not a store page that is then filtered down. > > > BasePersistence.listFullEntities already takes an entityFilter and > pages > > > after it. A dedicated store filter can come later if policy volume > needs > > > it. > > > > > > wdyt? > > > > > > Thanks, > > > Prithvi S > > > > > > On Sun, Sep 13, 2026 at 6:45 AM Yong Zheng <[email protected]> wrote: > > > > > > > Hello, > > > > > > > > I would like to start this ML for discussing if we should implement > the > > > > pagination support for policy APIs (GET policy only for this ML). > Here > > > is a > > > > reference report issue for this matter: > > > > https://github.com/apache/polaris/issues/5311. > > > > > > > > Currently we have following paths which should support pagination: > > > > > > > > 1. GET /polaris/v1/{prefix}/namespaces/{namespace}/policies > > > > 2. GET /polaris/v1/{prefix}/applicable-policies > > > > > > > > For GET policies, we are returning everything via > > > > PageToken.readEverything() ( > > > > > > > > https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/catalog/policy/PolicyCatalog.java#L177C15-L177C41 > > > > ) > > > > > > > > For GET applicable-policies, we are computing effective policy by > walking > > > > the entity hierarchy in memory rather than issuing an single > > > backing-store > > > > listing, we may need some discussion around this. But it may be > worth to > > > > start a different ML for this unless preferred by the community. > > > > > > > > Now to honor the pagination for GET policy, there are 2 cases which > we > > > > will need to support: > > > > > > > > case 1: No filter on policyType > > > > > > > > This is the simple case where we just need to thread the token and > > > > matching to listSemanticModels (ref: > > > > > > > > https://github.com/apache/polaris/blob/48126190377cfb2d0a54b4b9905a7375fde457ec/extensions/semantic-models/src/main/java/org/apache/polaris/service/catalog/semanticmodel/SemanticModelCatalogAdapter.java#L92 > > > > ). > > > > > > > > case 2: Filter on policyType > > > > > > > > When a policyType is been set, the current code is calling > > > > listFullEntitiesAll and perform filter in memory (ref: > > > > > > > > https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/catalog/policy/PolicyCatalog.java#L188 > > > > ). > > > > > > > > For case 2, I would like to see what community thinks regarding what > we > > > > should perform: > > > > 1. Ignore the pagination when a filter is set? > > > > - In this case, we stay with current behavior > > > > 2. Pagination at the post-filter? > > > > - We do pagination on the filtered response > > > > 3. Add a filter-aware store mthod? > > > > - Something likes listEntitiesByPolicyType that we handle pagination > and > > > > avoid current full fetch and in-memory filter > > > > > > > > Thanks, > > > > Yong > > > > > > > > > >
