himanshug commented on issue #6743: IncrementalIndex generally overestimates theta sketch size URL: https://github.com/apache/incubator-druid/issues/6743#issuecomment-486475596 > I wonder how bad it would be if aggregators were never allowed to free memory. Basically, let them allocate whatever chunks they want, and free it automatically at the end of the query, but don't let them free it. And if the total allocated memory exceeds the memory limit for a query, then fail the query. I intentionally left the detail of "pfree" implementation and I can imagine an interface for "MemoryAllocatorFactory" interface (which has factory method with a parameter for supplier of os memory e.g. StupidPool ). Implementation could be as simple as possible including free() being noop and a close() method on allocator that gets called at the end of query and all memory is freed at that point only. > We considered moving IncrementalIndex's aggregators off-heap out of scope at the time. I meant if allocator stuff was in place then we could use an heap-based-allocator that gets used from OnHeapIncrementalIndex . I think it would be simple, but then you never know till you prototype :)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
