+1. Handling pagination internally looks like a good move to me. It will help the CLI scale better and prevent massive API responses as the amount of data grows.
-Ayush On 2026/08/11 17:31:07 Yufei Gu wrote: > +1, I think supporting pagination internally in the CLI is reasonable. > > I agree with Yong that pagination and streaming output are separate > concerns. We can use pagination for listing requests to avoid a potentially > very large API response, while preserving the existing CLI output behavior. > This also gives us better scalability for large deployments without > introducing a user visible behavior change. > > Yufei > > > On Sat, Aug 8, 2026 at 12:55 PM Yong Zheng <[email protected]> wrote: > > > Hello Dmitri, > > > > I do agree that pagination is probably not a hard requirement for the > > typical one by one catalog management use case. My main concern is around > > avoiding a very large API response. Taking one example, for a shared > > tenants environment where we are creating two principals (e.g. RW and RO) > > per tenants, as the number of tenant increases (e.g. lets say 20k tenants), > > a single list principal would return 40k entries in a single response. I > > think streaming the results to STDOUT may be a diff question whether the > > CLI should use pagination or not internally for listing requests. We could > > use pagination to avoid a single large API response without changing the > > existing CLI output behavior. > > > > Thanks, > > Yong > > > > On 2026/08/08 00:13:47 Dmitri Bourlatchkov wrote: > > > Hi Yong, > > > > > > It would be nice to support pagination in the CLI from the completeness > > POV. > > > > > > However, I think this is largely a concern of practical CLI users. If the > > > amount of data people deal with is manageable in a single request, > > > pagination is probably not a hard requirement. > > > > > > I personally use the CLI only for one-by-one catalog management where the > > > amount of data is fairly small (create/update). > > > > > > If we add pagination, I think we will have to support proper unix-style > > > STDOUT streams to allow piping large amounts of data to other CLI tools. > > > This may have an impact on the modern colourful CLI output, but TBH, I'm > > > not very familiar with how it works ATM :) > > > > > > Cheers, > > > Dmitri. > > > > > > On Fri, Aug 7, 2026 at 2:14 AM Yong Zheng <[email protected]> wrote: > > > > > > > Hello, > > > > > > > > We have pageSize where "For servers that support pagination, this > > signals > > > > an upper bound of the number of results that a client will receive. For > > > > servers that do not support pagination, clients may receive results > > larger > > > > than the indicated pageSize", however, we are not take advantage of > > this. > > > > Currently all of the API used in CLI are straight full fetch which can > > be > > > > very expensive for large scale systems where the number of polaris > > entities > > > > are high (e.g. use unique principal for single tenant ingestions where > > the > > > > number of unique tenant is high). Should we consider adopting this > > setting > > > > in CLI? > > > > > > > > Thanks, > > > > Yong Zheng > > > > > > > > > >
