There actually is prior art on the partial data approach. That’s how scans work generally. MultiGets also return partial results when the throttle/limit is exceeded via MultiResultTooLargeException (which is immediately retried to get results for the rest of the batch). So this is a common pattern.
I think it’s very bad to sleep on the server side, so aggregating partial results on the client side seems like the cleanest approach. All of the aggregation client endpoints should be partially aggregatable on the client side. By that I mean it might be hard to return a partial result for an endpoint that returns a quantile, but thankfully all the endpoints return something that can be post aggregated. On Tue, Jul 16, 2024 at 6:57 PM Charles Connell <cconn...@hubspot.com.invalid> wrote: > Hi folks, > > I am considering contributing a PR that will allow the > AggregationClient+AggregateImplementation coprocessor to respect quota > throttling during its operations, or something in that spirit. I want > to gather input from the community on your thoughts about this. In > particular here are some questions: > > 1. Is this idea useful enough to be accepted as a contribution? > 2. Would you prefer a change to the existing coprocessor, or a new > coprocessor? > 3. One way to accomplish this is for the coprocessor endpoint to > return a partial result to the client each time it encounters an > RpcThrottleException. The client would then sleep and send a new > request, repeatedly, until the partial results cumulatively > represented the full scan it wanted. Another way to accomplish this is > for the coprocessor to sleep on the server side each time it > encounters an RpcThrottleException, and return a single complete > result. There are pros and cons to each of these approaches. I don't > believe there is prior art for either partial results or > server-side-sleeps in HBase, so both ideas feel awkward to me. What do > you think? > 4. If we go with the "partial results" idea, this seems hard to > accomplish in an async client. Does anyone have an example of how to > cleanly encapsulate a multi-request-response cycle of async requests > inside a CompletableFuture? I wonder if this is maybe not possible. > > Thanks for your time. > Charles > >
