Hello, I also think we should wait for a use-case for now. This can be a very expensive operation and we implicitly called out this in the CLI for why we are not exporting this due to the compute overhead.
Thanks, Yong On Fri, Oct 30, 2026 at 11:12 AM Yufei Gu <[email protected]> wrote: > > Yufei, which one you mean? > > Hi JB, I assume this is the question for Arnab. > > Yufei > > > On Fri, Oct 30, 2026 at 2:49 AM Jean-Baptiste Onofré <[email protected]> > wrote: > > > Hi guys, > > > > On Yufei's first point, I read "app-level pagination" in Arnab's proposal > > as service-layer pagination on the server (compute, sort, slice), not CLI > > pagination. > > Yufei, which one you mean? > > If you mean server-side, I agree that we should only do it for a real use > > case. > > > > About the questions: > > 1. I'm fine with app-level pagination as a v1. The endpoint already > > declares page-token, page-size and next-page-token in the spec, so > silently > > returning everything is a contract gap. I agree with Yufei "wait for a > use > > case" argument, because this version doesn't reduce server work for large > > hierarchies. I suggest to treat that as contract compliance rather than a > > scalability fix. If nobody has a concrete need soon, we could leave it > open > > :) > > 2. I agree with Yufei about the order. PolicyIdentifier (namespace > levels, > > then name) should already be unique. I would not add inheritable flag or > > source unless we find a real collision. We should document the order in > the > > spec so clients can rely on it. > > 3. No objection to not reuse the metastore PageToken encoding, since > there > > is no single store listing behind this call. I think the token must be > > opaque and self-validating, so we can have a clean 400 when the token is > > not correct (rather than an exception). Also, because the effective set > is > > recomputed on every request, I would encode the last-returned sort key > > rather than an offset. I believe it's more stable if policies are > attached > > or detached between pages. > > 4. I would keep this thread for the server side only. The CLI follow-up > can > > be a separate issue once the server returns next-page-token. > > > > Regards > > JB > > > > On Fri, Oct 30, 2026 at 2:11 AM Yufei Gu <[email protected]> wrote: > > > > > Hi Arnab, > > > > > > Thanks for picking this up. A few clarifications: > > > > > > 1. Does app-level pagination refer to CLI pagination? If so, I agree > > with > > > that approach. If it involves a server-side change, listPolicies and > > > listGenericTables would be good to have, but I suggest waiting until we > > > have a solid use case before moving forward with getApplicablePolicies. > > > > > > 2. Sorting by PolicyIdentifier (namespace, then name) should already > be > > > unique since the merge yields one entry per policy. Adding policy type > > as a > > > tie-breaker works as well. > > > > > > Best regards, > > > Yufei > > > > > > > > > On Wed, Oct 28, 2026 at 10:54 PM Arnab Karmakar <[email protected]> > > > wrote: > > > > > > > Hello, > > > > > > > > Following up on GitHub issue #5311 > > > > <https://github.com/apache/polaris/issues/5311> (policy / > > generic-table > > > > list pagination): listGenericTables (#5533 > > > > <https://github.com/apache/polaris/issues/5533>) and listPolicies > > (#5557 > > > > <https://github.com/apache/polaris/issues/5557>) are merged. The > only > > > > remaining gap is GET /polaris/v1/{prefix}/applicable-policies, which > > > > declares page-token, page-size, and next-page-token in the spec but > > still > > > > returns the full assembled result set. > > > > > > > > I plan to implement the remaining work and would like consensus on > how > > > > pagination should behave for this endpoint before opening a PR. > > > > > > > > Background > > > > ---------- > > > > Unlike listPolicies, getApplicablePolicies does not map to a single > > > > backing-store listing. PolicyCatalog#getEffectivePolicies walks the > > > entity > > > > path (catalog / namespace / table), loads policy mappings per level, > > > merges > > > > inheritable vs non-inheritable policies, and returns a > > > > List<ApplicablePolicy> that PolicyCatalogHandler currently wraps in a > > > > HashSet (no stable order, no continuation token). > > > > > > > > Proposal (for discussion) > > > > ------------------------- > > > > 1. Page boundary: After computing the full effective set for the > > request > > > > (same semantics as today), apply pagination in the service layer: > > > > sort into a deterministic order, return page-size items, and > expose > > > > next-page-token when more items remain. The page token would > encode > > > > an opaque cursor (e.g. offset into the sorted snapshot), not a > > > > metastore PageToken. > > > > > > > > This honors the REST contract and LIST_PAGINATION_MAX_PAGE_SIZE > > > > behavior without changing inheritance semantics. It does not > reduce > > > > server work for large hierarchies. > > > > > > > > 2. Stable ordering: Replace HashSet assembly with a sorted list > before > > > > slicing. Candidate sort key (lexicographic): > > > > policy identifier (namespace levels, then policy name), then > policy > > > > type, then inheritable flag / source if needed for uniqueness. > > > > The response schema keeps uniqueItems. > > > > > > > > 3. Adapter/handler: Thread pageToken and pageSize through > > > > PolicyCatalogAdapter#getApplicablePolicies into > > > > PolicyCatalogHandler#getApplicablePolicies, mirroring listPolicies > > > > (PageToken.build, rejectIncompleteListing, next-page-token on the > > > > response). > > > > > > > > Questions for the list > > > > ---------------------- > > > > 1. Is application-level pagination (full compute + sort + slice) an > > > > acceptable v1 for this endpoint, given there is no single store > listing > > > to > > > > paginate? > > > > > > > > 2. What total order should clients rely on across pages? Is sorting > > > > by PolicyIdentifier (namespace + name) then policy type sufficient, > or > > > > should we document a different key? > > > > > > > > 3. Any objection to encoding the continuation as an opaque in-memory > > > cursor > > > > rather than reusing metastore PageToken encoding? > > > > > > > > 4. Scope: Should this thread cover only applicable-policies, or also > > any > > > > CLI follow-up (#5385-style) once the server returns next-page-token? > > > > > > > > Please let me know if this direction sounds reasonable. > > > > > > > > Thanks, > > > > Arnab > > > > > > > > > >
