Hi Rishi,

Thanks for raising this here. I am +1 on the direction.

Thanks for driving this change, Dmitri. I did another round of review, and
it looks good to me overall, aside from a few minor wording adjustments.

Yufei


On Tue, Nov 17, 2026 at 2:58 PM Dmitri Bourlatchkov via dev <
[email protected]> wrote:

> Hi JB,
>
> > I would state in the spec that an empty pageToken means the
> > same as no pageToken.
>
> Good point! +1
>
> > LIST_PAGINATION_MAX_PAGE_SIZE is unlimited by default [...]
>
> Also a very good point!
>
> In practice I think LIST_PAGINATION_MAX_PAGE_SIZE applies to the IRC API
> and acts as a safety valve to be configured by Polaris users.
>
> I think new APIs that use Rishi's (and compatible) spec wording probably
> need a _different_ config for limiting page sizes, and it has to be ON by
> default.
>
> I think this can be resolved in implementation PRs. The spec seems to be
> unaffected by this.
>
> I believe Anand already took this approach in [4115].
>
> > Regarding getApplicablePolicies [...]
>
> Let's have a separate discussion for that. I believe we cannot break this
> API contract now that we have released it without any "beta" or
> "experimental" annotations... that is per our evolutions guidelines [1]
>
> Cheers,
> Dmitri.
>
> [1] https://polaris.apache.org/releases/1.9.0/evolution/
>
> [4115] https://github.com/apache/polaris/pull/4115
>
> On Sat, Nov 14, 2026 at 2:33 AM Jean-Baptiste Onofré <[email protected]>
> wrote:
>
> > Hi Rishi,
> >
> > I'm +1 on the approach. I don't think it conflicts with what we agreed in
> > September. Fail-closed protects clients written against the Iceberg
> promise
> > that a request with no pageToken gets the complete result. A new Polaris
> > native operation makes no such promise, so "always follow the token" is
> the
> > cleaner contract, and it bounds response size from day one. One shared
> > definition is also better than a copy per operation.
> >
> > We had a similar discussion about pagination in the Directory Spec PR.
> >
> > I have a few comments (nothing blocking PR #6028):
> > - As Prithvi noted in the September thread, Java Iceberg clients send an
> > empty pageToken to start paginating. These operations always allow
> > pagination, so I would state in the spec that an empty pageToken means
> the
> > same as no pageToken. Then a client with Iceberg habits doesn't get a
> 400.
> > - The storage configuration listings have filters (targetExists,
> > referenced). It would be good to say that a client must send the same
> > parameters on every page, apart from pageToken and pageSize. The spec
> could
> > also say the server may reject a token that comes with different
> > parameters.
> > - LIST_PAGINATION_MAX_PAGE_SIZE is unlimited by default. If the new
> > operations paginate only when an operator sets a maximum, a client that
> > ignores the token works in every default deployment. Then it silently
> loses
> > results the day someone sets a limit :) That's the September problem
> again,
> > just moved :) I would suggest that these operations use a reasonable
> > default page size even when no maximum is set, so clients use the token
> > from the start. This belongs to the impl PRs after #5943, not in #6028.
> > - #5533 showed that the generic tables CLI followed next-page-token only
> > when --page-size was set. When the storage configuration operations land,
> > the Python CLI should always follow the token.
> >
> > Regarding getApplicablePolicies, I would keep the Iceberg rules and fix
> > #5311 under the current contract, the same as listPolicies. Putting one
> > policy operation under different rules would be confusing, and it would
> > change the contract for existing clients. If we ever want to move all the
> > Polaris catalog listings (policies, generic tables, semantic models, ...)
> > to these definitions, that should be its own discussion covering all of
> > them.
> >
> > The Shares and Directory APIs already behave this way, so switching it to
> > reference the new file could be a follow-up with no behavior change.
> >
> > Regards
> > JB
> >
> > On Sat, Nov 14, 2026 at 2:18 AM Srinivas Rishindra <
> [email protected]
> > >
> > wrote:
> >
> > > Hi all,
> > >
> > > PR #6028 [1] adds spec/polaris-pagination.yaml, a small set of
> pagination
> > > definitions for Polaris-specific list operations: a pageToken and a
> > > pageSize
> > >  query parameter, and a page token schema. Please give explicit
> sign-off
> > > here before any operation starts using it, because the rules differ
> from
> > > the Iceberg REST Catalog API. I'd like to settle that.
> > >
> > > The file states four rules:
> > >
> > >    1. Any response of an operation that uses these definitions may be
> > >    paginated. A client must always check for a page token in the
> response
> > > and,
> > >    when one is present, send it unchanged as the pageToken query
> > parameter
> > >    to get the next page. The server omits the page token from the last
> > > page. A
> > >    client must treat a missing or null page token as the end of the
> > > results.
> > >    2. A client does not need to send pageToken or pageSize to start
> > >    paginating. The server may paginate a response to a request that has
> > >    neither.
> > >    3. The server decides the page size. A page may hold fewer results
> > than
> > >    pageSize.
> > >    4. A server that does not paginate returns all results in one
> > response,
> > >    with no page token.
> > >
> > > Scope: the rules apply only to operations that reference this file.
> > Nothing
> > > on main references it today. Operations that use the Iceberg REST
> Catalog
> > > pagination definitions keep the Iceberg rules, including the Polaris
> > > catalog listings that reuse them (policies, generic tables, semantic
> > > models)
> > > .
> > >
> > > How this relates to the September thread [2] and #5553 [3]: that thread
> > > found that a configured LIST_PAGINATION_MAX_PAGE_SIZE truncated
> requests
> > > with no pageToken, while the Iceberg spec promises such a request the
> > > complete result. We agreed to reject those requests rather than
> truncate
> > > them, and #5553 did that. The concern there was clients that rely on
> > > the Iceberg
> > > promise and never look for a token. These rules don't touch those
> > > operations. For a new operation that uses this file, no such clients
> > > exist: the
> > > contract tells every client from day one to follow the token, so a
> first
> > > page is never mistaken for the whole result. It also gives the server
> the
> > > bound on response size that motivated the max page size in the first
> > place.
> > >
> > > There is precedent: the Shares API spec on main (
> > > spec/polaris-shares-api.yaml) already says that when pageSize is absent
> > the
> > > service's maximum is used, and that nextPageToken is present when more
> > > results exist. This file writes that behavior down once for reuse.
> > >
> > > First intended users: the two list operations in the storage
> > configuration
> > > spec, PR #5943 [4] (listStorageConfigurations and listStorageMappings,
> in
> > > the management API).
> > >
> > > One related question: getApplicablePolicies declares the Iceberg
> > pagination
> > > parameters but doesn't paginate yet (#5311 [5]). Should it keep the
> > Iceberg
> > > rules when that is fixed, like listPolicies next to it, or move to
> these
> > > definitions? Moving it would change the contract of an existing
> > operation,
> > > so
> > > I'd treat it as a separate decision either way.
> > >
> > > Please reply with +1 or any objections. If there are none in about 72
> > > hours,
> > >  I'll take that as agreement and ask for #6028 to be merged.
> > >
> > > [1] https://github.com/apache/polaris/pull/6028 [2]
> > > https://lists.apache.org/thread/k81ptyktdbdf8gynncgk3o04mqt85zyk [3]
> > > https://github.com/apache/polaris/pull/5553 [4]
> > > https://github.com/apache/polaris/pull/5943 [5]
> > > https://github.com/apache/polaris/issues/5311
> > >
> > > Thanks,
> > > Rishi
> > >
> >
>

Reply via email to