Hi all,

+1 on #6028 and the four rules as scoped.

This does not conflict with the September thread or #5553. Fail-closed
there protects clients that rely on the Iceberg promise that a request
without a pageToken returns the complete result. An operation that
opts into polaris-pagination.yaml has a different contract from day
one, where every client follows the token, so there are no such
clients to protect. Writing the Shares API behaviour down once for
reuse makes sense.

One implementation note on the helper. rejectIncompleteListing encodes
the Iceberg unsized rule: it rejects only when pageToken and pageSize
are both absent and a next token comes back. Under these rules that
case is a valid first page, so reusing it on a Polaris-pagination
listing would turn a correct response into a 400. It is protected on
CatalogHandler, and the only subclasses are the Iceberg,
generic-table, policy and semantic-model handlers, so the management
API operations in #5943 cannot reach it. The case to watch is a future
catalog-side operation that adopts these rules, where it is inherited
and the mistake is easy to make. Such an operation needs a path that
allows a token on the first page.

On getApplicablePolicies, I would keep the Iceberg rules when
pagination lands, matching listPolicies beside it. It already
advertises those parameters, so moving it would change the contract of
an existing operation. Worth deciding separately when #5311 is
implemented.

-Ayush

On Sun, 15 Nov 2026 at 03:23, vignesh a <[email protected]> wrote:
>
> Hi all,
>
> +1 on #6028 from me. I think new Polaris list operations
> should follow the token from the first response on the next
> request, and the Iceberg complete-result promise stays where
> it is, on the Iceberg operations (#5553 included).
>
> One thing to watch in the impl. Right now
> CatalogHandler.rejectIncompleteListing only rejects when
> both pageToken and pageSize are missing and a next token
> comes back [1]. That's really the Iceberg unsized rule. For
> these new operations the first response is allowed to carry
> a next-page token, and a missing or empty token both just
> mean "first page". So if #5943 calls that helper, it'd 400
> a first page that's actually valid under these rules.
>
> Thanks,
> Vignesh
>
> [1]
> https://github.com/apache/polaris/blob/67d10ae1629927fbac048cd4aa09a5906a1cd5dd
> /runtime/service/src/main/java/org/apache/polaris
> /service/catalog/common/CatalogHandler.java#L124
>
> On Sat, 14 Nov 2026 at 13:02, Jean-Baptiste Onofré <[email protected]> wrote:
>
> > Hi Rishi,
> >
> > I'm +1 on the approach. I don't think it conflicts with what we agreed in
> > September. Fail-closed protects clients written against the Iceberg promise
> > that a request with no pageToken gets the complete result. A new Polaris
> > native operation makes no such promise, so "always follow the token" is the
> > cleaner contract, and it bounds response size from day one. One shared
> > definition is also better than a copy per operation.
> >
> > We had a similar discussion about pagination in the Directory Spec PR.
> >
> > I have a few comments (nothing blocking PR #6028):
> > - As Prithvi noted in the September thread, Java Iceberg clients send an
> > empty pageToken to start paginating. These operations always allow
> > pagination, so I would state in the spec that an empty pageToken means the
> > same as no pageToken. Then a client with Iceberg habits doesn't get a 400.
> > - The storage configuration listings have filters (targetExists,
> > referenced). It would be good to say that a client must send the same
> > parameters on every page, apart from pageToken and pageSize. The spec could
> > also say the server may reject a token that comes with different
> > parameters.
> > - LIST_PAGINATION_MAX_PAGE_SIZE is unlimited by default. If the new
> > operations paginate only when an operator sets a maximum, a client that
> > ignores the token works in every default deployment. Then it silently loses
> > results the day someone sets a limit :) That's the September problem again,
> > just moved :) I would suggest that these operations use a reasonable
> > default page size even when no maximum is set, so clients use the token
> > from the start. This belongs to the impl PRs after #5943, not in #6028.
> > - #5533 showed that the generic tables CLI followed next-page-token only
> > when --page-size was set. When the storage configuration operations land,
> > the Python CLI should always follow the token.
> >
> > Regarding getApplicablePolicies, I would keep the Iceberg rules and fix
> > #5311 under the current contract, the same as listPolicies. Putting one
> > policy operation under different rules would be confusing, and it would
> > change the contract for existing clients. If we ever want to move all the
> > Polaris catalog listings (policies, generic tables, semantic models, ...)
> > to these definitions, that should be its own discussion covering all of
> > them.
> >
> > The Shares and Directory APIs already behave this way, so switching it to
> > reference the new file could be a follow-up with no behavior change.
> >
> > Regards
> > JB
> >
> > On Sat, Nov 14, 2026 at 2:18 AM Srinivas Rishindra <[email protected]
> > >
> > wrote:
> >
> > > Hi all,
> > >
> > > PR #6028 [1] adds spec/polaris-pagination.yaml, a small set of pagination
> > > definitions for Polaris-specific list operations: a pageToken and a
> > > pageSize
> > >  query parameter, and a page token schema. Please give explicit sign-off
> > > here before any operation starts using it, because the rules differ from
> > > the Iceberg REST Catalog API. I'd like to settle that.
> > >
> > > The file states four rules:
> > >
> > >    1. Any response of an operation that uses these definitions may be
> > >    paginated. A client must always check for a page token in the response
> > > and,
> > >    when one is present, send it unchanged as the pageToken query
> > parameter
> > >    to get the next page. The server omits the page token from the last
> > > page. A
> > >    client must treat a missing or null page token as the end of the
> > > results.
> > >    2. A client does not need to send pageToken or pageSize to start
> > >    paginating. The server may paginate a response to a request that has
> > >    neither.
> > >    3. The server decides the page size. A page may hold fewer results
> > than
> > >    pageSize.
> > >    4. A server that does not paginate returns all results in one
> > response,
> > >    with no page token.
> > >
> > > Scope: the rules apply only to operations that reference this file.
> > Nothing
> > > on main references it today. Operations that use the Iceberg REST Catalog
> > > pagination definitions keep the Iceberg rules, including the Polaris
> > > catalog listings that reuse them (policies, generic tables, semantic
> > > models)
> > > .
> > >
> > > How this relates to the September thread [2] and #5553 [3]: that thread
> > > found that a configured LIST_PAGINATION_MAX_PAGE_SIZE truncated requests
> > > with no pageToken, while the Iceberg spec promises such a request the
> > > complete result. We agreed to reject those requests rather than truncate
> > > them, and #5553 did that. The concern there was clients that rely on
> > > the Iceberg
> > > promise and never look for a token. These rules don't touch those
> > > operations. For a new operation that uses this file, no such clients
> > > exist: the
> > > contract tells every client from day one to follow the token, so a first
> > > page is never mistaken for the whole result. It also gives the server the
> > > bound on response size that motivated the max page size in the first
> > place.
> > >
> > > There is precedent: the Shares API spec on main (
> > > spec/polaris-shares-api.yaml) already says that when pageSize is absent
> > the
> > > service's maximum is used, and that nextPageToken is present when more
> > > results exist. This file writes that behavior down once for reuse.
> > >
> > > First intended users: the two list operations in the storage
> > configuration
> > > spec, PR #5943 [4] (listStorageConfigurations and listStorageMappings, in
> > > the management API).
> > >
> > > One related question: getApplicablePolicies declares the Iceberg
> > pagination
> > > parameters but doesn't paginate yet (#5311 [5]). Should it keep the
> > Iceberg
> > > rules when that is fixed, like listPolicies next to it, or move to these
> > > definitions? Moving it would change the contract of an existing
> > operation,
> > > so
> > > I'd treat it as a separate decision either way.
> > >
> > > Please reply with +1 or any objections. If there are none in about 72
> > > hours,
> > >  I'll take that as agreement and ask for #6028 to be merged.
> > >
> > > [1] https://github.com/apache/polaris/pull/6028 [2]
> > > https://lists.apache.org/thread/k81ptyktdbdf8gynncgk3o04mqt85zyk [3]
> > > https://github.com/apache/polaris/pull/5553 [4]
> > > https://github.com/apache/polaris/pull/5943 [5]
> > > https://github.com/apache/polaris/issues/5311
> > >
> > > Thanks,
> > > Rishi
> > >
> >

Reply via email to