gianm commented on issue #6743: IncrementalIndex generally overestimates theta sketch size URL: https://github.com/apache/incubator-druid/issues/6743#issuecomment-448030739 @AlexanderSaydakov One weird thing about how the ingestion-time row size estimator works is that it's using on-heap Aggregators, but it estimates size via the AggregatorFactory `getMaxIntermediateSize()` method that is meant to be the max size of a buffer needed for the BufferAggregator. One thing we could do is add a method to Aggregator (the on-heap interface) that estimates the current on-heap footprint. It can vary in a way that BufferAggregator, currently, can't. Another thing we could do is implement growable BufferAggregators and switch IncrementalIndex to use those instead. It has come up before, see proposal from @himanshug in https://github.com/apache/incubator-druid/issues/2963, but isn't implemented.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
