> Yufei, which one you mean? Hi JB, I assume this is the question for Arnab.
Yufei On Fri, Oct 30, 2026 at 2:49 AM Jean-Baptiste Onofré <[email protected]> wrote: > Hi guys, > > On Yufei's first point, I read "app-level pagination" in Arnab's proposal > as service-layer pagination on the server (compute, sort, slice), not CLI > pagination. > Yufei, which one you mean? > If you mean server-side, I agree that we should only do it for a real use > case. > > About the questions: > 1. I'm fine with app-level pagination as a v1. The endpoint already > declares page-token, page-size and next-page-token in the spec, so silently > returning everything is a contract gap. I agree with Yufei "wait for a use > case" argument, because this version doesn't reduce server work for large > hierarchies. I suggest to treat that as contract compliance rather than a > scalability fix. If nobody has a concrete need soon, we could leave it open > :) > 2. I agree with Yufei about the order. PolicyIdentifier (namespace levels, > then name) should already be unique. I would not add inheritable flag or > source unless we find a real collision. We should document the order in the > spec so clients can rely on it. > 3. No objection to not reuse the metastore PageToken encoding, since there > is no single store listing behind this call. I think the token must be > opaque and self-validating, so we can have a clean 400 when the token is > not correct (rather than an exception). Also, because the effective set is > recomputed on every request, I would encode the last-returned sort key > rather than an offset. I believe it's more stable if policies are attached > or detached between pages. > 4. I would keep this thread for the server side only. The CLI follow-up can > be a separate issue once the server returns next-page-token. > > Regards > JB > > On Fri, Oct 30, 2026 at 2:11 AM Yufei Gu <[email protected]> wrote: > > > Hi Arnab, > > > > Thanks for picking this up. A few clarifications: > > > > 1. Does app-level pagination refer to CLI pagination? If so, I agree > with > > that approach. If it involves a server-side change, listPolicies and > > listGenericTables would be good to have, but I suggest waiting until we > > have a solid use case before moving forward with getApplicablePolicies. > > > > 2. Sorting by PolicyIdentifier (namespace, then name) should already be > > unique since the merge yields one entry per policy. Adding policy type > as a > > tie-breaker works as well. > > > > Best regards, > > Yufei > > > > > > On Wed, Oct 28, 2026 at 10:54 PM Arnab Karmakar <[email protected]> > > wrote: > > > > > Hello, > > > > > > Following up on GitHub issue #5311 > > > <https://github.com/apache/polaris/issues/5311> (policy / > generic-table > > > list pagination): listGenericTables (#5533 > > > <https://github.com/apache/polaris/issues/5533>) and listPolicies > (#5557 > > > <https://github.com/apache/polaris/issues/5557>) are merged. The only > > > remaining gap is GET /polaris/v1/{prefix}/applicable-policies, which > > > declares page-token, page-size, and next-page-token in the spec but > still > > > returns the full assembled result set. > > > > > > I plan to implement the remaining work and would like consensus on how > > > pagination should behave for this endpoint before opening a PR. > > > > > > Background > > > ---------- > > > Unlike listPolicies, getApplicablePolicies does not map to a single > > > backing-store listing. PolicyCatalog#getEffectivePolicies walks the > > entity > > > path (catalog / namespace / table), loads policy mappings per level, > > merges > > > inheritable vs non-inheritable policies, and returns a > > > List<ApplicablePolicy> that PolicyCatalogHandler currently wraps in a > > > HashSet (no stable order, no continuation token). > > > > > > Proposal (for discussion) > > > ------------------------- > > > 1. Page boundary: After computing the full effective set for the > request > > > (same semantics as today), apply pagination in the service layer: > > > sort into a deterministic order, return page-size items, and expose > > > next-page-token when more items remain. The page token would encode > > > an opaque cursor (e.g. offset into the sorted snapshot), not a > > > metastore PageToken. > > > > > > This honors the REST contract and LIST_PAGINATION_MAX_PAGE_SIZE > > > behavior without changing inheritance semantics. It does not reduce > > > server work for large hierarchies. > > > > > > 2. Stable ordering: Replace HashSet assembly with a sorted list before > > > slicing. Candidate sort key (lexicographic): > > > policy identifier (namespace levels, then policy name), then policy > > > type, then inheritable flag / source if needed for uniqueness. > > > The response schema keeps uniqueItems. > > > > > > 3. Adapter/handler: Thread pageToken and pageSize through > > > PolicyCatalogAdapter#getApplicablePolicies into > > > PolicyCatalogHandler#getApplicablePolicies, mirroring listPolicies > > > (PageToken.build, rejectIncompleteListing, next-page-token on the > > > response). > > > > > > Questions for the list > > > ---------------------- > > > 1. Is application-level pagination (full compute + sort + slice) an > > > acceptable v1 for this endpoint, given there is no single store listing > > to > > > paginate? > > > > > > 2. What total order should clients rely on across pages? Is sorting > > > by PolicyIdentifier (namespace + name) then policy type sufficient, or > > > should we document a different key? > > > > > > 3. Any objection to encoding the continuation as an opaque in-memory > > cursor > > > rather than reusing metastore PageToken encoding? > > > > > > 4. Scope: Should this thread cover only applicable-policies, or also > any > > > CLI follow-up (#5385-style) once the server returns next-page-token? > > > > > > Please let me know if this direction sounds reasonable. > > > > > > Thanks, > > > Arnab > > > > > >
