Hi Rishi, Thanks for raising this here. I am +1 on the direction.
Thanks for driving this change, Dmitri. I did another round of review, and it looks good to me overall, aside from a few minor wording adjustments. Yufei On Tue, Nov 17, 2026 at 2:58 PM Dmitri Bourlatchkov via dev < [email protected]> wrote: > Hi JB, > > > I would state in the spec that an empty pageToken means the > > same as no pageToken. > > Good point! +1 > > > LIST_PAGINATION_MAX_PAGE_SIZE is unlimited by default [...] > > Also a very good point! > > In practice I think LIST_PAGINATION_MAX_PAGE_SIZE applies to the IRC API > and acts as a safety valve to be configured by Polaris users. > > I think new APIs that use Rishi's (and compatible) spec wording probably > need a _different_ config for limiting page sizes, and it has to be ON by > default. > > I think this can be resolved in implementation PRs. The spec seems to be > unaffected by this. > > I believe Anand already took this approach in [4115]. > > > Regarding getApplicablePolicies [...] > > Let's have a separate discussion for that. I believe we cannot break this > API contract now that we have released it without any "beta" or > "experimental" annotations... that is per our evolutions guidelines [1] > > Cheers, > Dmitri. > > [1] https://polaris.apache.org/releases/1.9.0/evolution/ > > [4115] https://github.com/apache/polaris/pull/4115 > > On Sat, Nov 14, 2026 at 2:33 AM Jean-Baptiste Onofré <[email protected]> > wrote: > > > Hi Rishi, > > > > I'm +1 on the approach. I don't think it conflicts with what we agreed in > > September. Fail-closed protects clients written against the Iceberg > promise > > that a request with no pageToken gets the complete result. A new Polaris > > native operation makes no such promise, so "always follow the token" is > the > > cleaner contract, and it bounds response size from day one. One shared > > definition is also better than a copy per operation. > > > > We had a similar discussion about pagination in the Directory Spec PR. > > > > I have a few comments (nothing blocking PR #6028): > > - As Prithvi noted in the September thread, Java Iceberg clients send an > > empty pageToken to start paginating. These operations always allow > > pagination, so I would state in the spec that an empty pageToken means > the > > same as no pageToken. Then a client with Iceberg habits doesn't get a > 400. > > - The storage configuration listings have filters (targetExists, > > referenced). It would be good to say that a client must send the same > > parameters on every page, apart from pageToken and pageSize. The spec > could > > also say the server may reject a token that comes with different > > parameters. > > - LIST_PAGINATION_MAX_PAGE_SIZE is unlimited by default. If the new > > operations paginate only when an operator sets a maximum, a client that > > ignores the token works in every default deployment. Then it silently > loses > > results the day someone sets a limit :) That's the September problem > again, > > just moved :) I would suggest that these operations use a reasonable > > default page size even when no maximum is set, so clients use the token > > from the start. This belongs to the impl PRs after #5943, not in #6028. > > - #5533 showed that the generic tables CLI followed next-page-token only > > when --page-size was set. When the storage configuration operations land, > > the Python CLI should always follow the token. > > > > Regarding getApplicablePolicies, I would keep the Iceberg rules and fix > > #5311 under the current contract, the same as listPolicies. Putting one > > policy operation under different rules would be confusing, and it would > > change the contract for existing clients. If we ever want to move all the > > Polaris catalog listings (policies, generic tables, semantic models, ...) > > to these definitions, that should be its own discussion covering all of > > them. > > > > The Shares and Directory APIs already behave this way, so switching it to > > reference the new file could be a follow-up with no behavior change. > > > > Regards > > JB > > > > On Sat, Nov 14, 2026 at 2:18 AM Srinivas Rishindra < > [email protected] > > > > > wrote: > > > > > Hi all, > > > > > > PR #6028 [1] adds spec/polaris-pagination.yaml, a small set of > pagination > > > definitions for Polaris-specific list operations: a pageToken and a > > > pageSize > > > query parameter, and a page token schema. Please give explicit > sign-off > > > here before any operation starts using it, because the rules differ > from > > > the Iceberg REST Catalog API. I'd like to settle that. > > > > > > The file states four rules: > > > > > > 1. Any response of an operation that uses these definitions may be > > > paginated. A client must always check for a page token in the > response > > > and, > > > when one is present, send it unchanged as the pageToken query > > parameter > > > to get the next page. The server omits the page token from the last > > > page. A > > > client must treat a missing or null page token as the end of the > > > results. > > > 2. A client does not need to send pageToken or pageSize to start > > > paginating. The server may paginate a response to a request that has > > > neither. > > > 3. The server decides the page size. A page may hold fewer results > > than > > > pageSize. > > > 4. A server that does not paginate returns all results in one > > response, > > > with no page token. > > > > > > Scope: the rules apply only to operations that reference this file. > > Nothing > > > on main references it today. Operations that use the Iceberg REST > Catalog > > > pagination definitions keep the Iceberg rules, including the Polaris > > > catalog listings that reuse them (policies, generic tables, semantic > > > models) > > > . > > > > > > How this relates to the September thread [2] and #5553 [3]: that thread > > > found that a configured LIST_PAGINATION_MAX_PAGE_SIZE truncated > requests > > > with no pageToken, while the Iceberg spec promises such a request the > > > complete result. We agreed to reject those requests rather than > truncate > > > them, and #5553 did that. The concern there was clients that rely on > > > the Iceberg > > > promise and never look for a token. These rules don't touch those > > > operations. For a new operation that uses this file, no such clients > > > exist: the > > > contract tells every client from day one to follow the token, so a > first > > > page is never mistaken for the whole result. It also gives the server > the > > > bound on response size that motivated the max page size in the first > > place. > > > > > > There is precedent: the Shares API spec on main ( > > > spec/polaris-shares-api.yaml) already says that when pageSize is absent > > the > > > service's maximum is used, and that nextPageToken is present when more > > > results exist. This file writes that behavior down once for reuse. > > > > > > First intended users: the two list operations in the storage > > configuration > > > spec, PR #5943 [4] (listStorageConfigurations and listStorageMappings, > in > > > the management API). > > > > > > One related question: getApplicablePolicies declares the Iceberg > > pagination > > > parameters but doesn't paginate yet (#5311 [5]). Should it keep the > > Iceberg > > > rules when that is fixed, like listPolicies next to it, or move to > these > > > definitions? Moving it would change the contract of an existing > > operation, > > > so > > > I'd treat it as a separate decision either way. > > > > > > Please reply with +1 or any objections. If there are none in about 72 > > > hours, > > > I'll take that as agreement and ask for #6028 to be merged. > > > > > > [1] https://github.com/apache/polaris/pull/6028 [2] > > > https://lists.apache.org/thread/k81ptyktdbdf8gynncgk3o04mqt85zyk [3] > > > https://github.com/apache/polaris/pull/5553 [4] > > > https://github.com/apache/polaris/pull/5943 [5] > > > https://github.com/apache/polaris/issues/5311 > > > > > > Thanks, > > > Rishi > > > > > >
