Hi Rishi, I'm +1 on the approach. I don't think it conflicts with what we agreed in September. Fail-closed protects clients written against the Iceberg promise that a request with no pageToken gets the complete result. A new Polaris native operation makes no such promise, so "always follow the token" is the cleaner contract, and it bounds response size from day one. One shared definition is also better than a copy per operation.
We had a similar discussion about pagination in the Directory Spec PR. I have a few comments (nothing blocking PR #6028): - As Prithvi noted in the September thread, Java Iceberg clients send an empty pageToken to start paginating. These operations always allow pagination, so I would state in the spec that an empty pageToken means the same as no pageToken. Then a client with Iceberg habits doesn't get a 400. - The storage configuration listings have filters (targetExists, referenced). It would be good to say that a client must send the same parameters on every page, apart from pageToken and pageSize. The spec could also say the server may reject a token that comes with different parameters. - LIST_PAGINATION_MAX_PAGE_SIZE is unlimited by default. If the new operations paginate only when an operator sets a maximum, a client that ignores the token works in every default deployment. Then it silently loses results the day someone sets a limit :) That's the September problem again, just moved :) I would suggest that these operations use a reasonable default page size even when no maximum is set, so clients use the token from the start. This belongs to the impl PRs after #5943, not in #6028. - #5533 showed that the generic tables CLI followed next-page-token only when --page-size was set. When the storage configuration operations land, the Python CLI should always follow the token. Regarding getApplicablePolicies, I would keep the Iceberg rules and fix #5311 under the current contract, the same as listPolicies. Putting one policy operation under different rules would be confusing, and it would change the contract for existing clients. If we ever want to move all the Polaris catalog listings (policies, generic tables, semantic models, ...) to these definitions, that should be its own discussion covering all of them. The Shares and Directory APIs already behave this way, so switching it to reference the new file could be a follow-up with no behavior change. Regards JB On Sat, Nov 14, 2026 at 2:18 AM Srinivas Rishindra <[email protected]> wrote: > Hi all, > > PR #6028 [1] adds spec/polaris-pagination.yaml, a small set of pagination > definitions for Polaris-specific list operations: a pageToken and a > pageSize > query parameter, and a page token schema. Please give explicit sign-off > here before any operation starts using it, because the rules differ from > the Iceberg REST Catalog API. I'd like to settle that. > > The file states four rules: > > 1. Any response of an operation that uses these definitions may be > paginated. A client must always check for a page token in the response > and, > when one is present, send it unchanged as the pageToken query parameter > to get the next page. The server omits the page token from the last > page. A > client must treat a missing or null page token as the end of the > results. > 2. A client does not need to send pageToken or pageSize to start > paginating. The server may paginate a response to a request that has > neither. > 3. The server decides the page size. A page may hold fewer results than > pageSize. > 4. A server that does not paginate returns all results in one response, > with no page token. > > Scope: the rules apply only to operations that reference this file. Nothing > on main references it today. Operations that use the Iceberg REST Catalog > pagination definitions keep the Iceberg rules, including the Polaris > catalog listings that reuse them (policies, generic tables, semantic > models) > . > > How this relates to the September thread [2] and #5553 [3]: that thread > found that a configured LIST_PAGINATION_MAX_PAGE_SIZE truncated requests > with no pageToken, while the Iceberg spec promises such a request the > complete result. We agreed to reject those requests rather than truncate > them, and #5553 did that. The concern there was clients that rely on > the Iceberg > promise and never look for a token. These rules don't touch those > operations. For a new operation that uses this file, no such clients > exist: the > contract tells every client from day one to follow the token, so a first > page is never mistaken for the whole result. It also gives the server the > bound on response size that motivated the max page size in the first place. > > There is precedent: the Shares API spec on main ( > spec/polaris-shares-api.yaml) already says that when pageSize is absent the > service's maximum is used, and that nextPageToken is present when more > results exist. This file writes that behavior down once for reuse. > > First intended users: the two list operations in the storage configuration > spec, PR #5943 [4] (listStorageConfigurations and listStorageMappings, in > the management API). > > One related question: getApplicablePolicies declares the Iceberg pagination > parameters but doesn't paginate yet (#5311 [5]). Should it keep the Iceberg > rules when that is fixed, like listPolicies next to it, or move to these > definitions? Moving it would change the contract of an existing operation, > so > I'd treat it as a separate decision either way. > > Please reply with +1 or any objections. If there are none in about 72 > hours, > I'll take that as agreement and ask for #6028 to be merged. > > [1] https://github.com/apache/polaris/pull/6028 [2] > https://lists.apache.org/thread/k81ptyktdbdf8gynncgk3o04mqt85zyk [3] > https://github.com/apache/polaris/pull/5553 [4] > https://github.com/apache/polaris/pull/5943 [5] > https://github.com/apache/polaris/issues/5311 > > Thanks, > Rishi >
