kfaraz commented on issue #12022: URL: https://github.com/apache/druid/issues/12022#issuecomment-986788236
Based on the discussion above, we will modify the aggregator interfaces as follows: Interface `AggregatorFactory` will get a new `factorizeWithSize()` method which returns a structure containing both the `Aggregator` instance as well as the initial memory size. Interface `Aggregator` will get a new `aggregateWithSize()` method which returns a long representing the incremental memory used in that invocation of `aggregateWithSize()`. The default impl of this method would call `aggregate()` and return 0. Aggregators such as `sum` can rely on the default impl itself, thus always returning 0 from `aggregateWithSize()` effectively making the aggregator size same as the initial size returned from `AggregatorFactory.factorizeWithSize()` In the first iteration, we would put the new estimation logic behind a feature flag. After some more testing, the flag can be removed altogether. The following metrics will also be added (if they don't already exist): - number of incremental persists - num rows/num bytes in each persist - time taken to fill up buffer - time taken to persist buffer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
