gianm commented on issue #6743: IncrementalIndex generally overestimates theta sketch size URL: https://github.com/apache/incubator-druid/issues/6743#issuecomment-451254639 > I think you are referring to the HyperLogLog (HLL) sketch implementation that was developed for Druid some years ago. There are several things you need to understand about that implementation. @leerho, yep, you are right, I am talking about that one. I am aware of the things you mentioned and appreciate the work you've done in the characterization studies (I was not aware of the problems with our core HLL implementation until that work was published). I only mentioned it by way of explaining why we never worried about growability in the past, back when that was the only sketch we supported. It's a separate issue from this one, but, I'd like to take you up on working out a way to migrate users to the new sketch. It is somewhat delicate since the new one is in an extension and the old one is in core. And also because they don't have the same on-disk format, so we'd need to support both forever (we don't want to drop compatibility with on-disk data). We need to get the migration right to avoid user confusion. > I'm not sure I follow your proposal and may be missing something. What part is unclear? I can tell from your other comments that you are following at least the main idea of it: that each aggregation operator would get a chunk of memory and be responsible for using it to store sketch data for all the tuples being aggregated for a given segment. > Nonetheless, one of the challenges of just carving out a chunk of the processing buffer to be managed by by the individual BufferAggregators is that in order truly gain back the "unused memory" would require memory management sophistication similar to the design of a malloc(), free() and a garbage collector, which is non-trivial. Imagine each sketch with an internal hash-table. As individual sketches grow they need to obtain a larger chunk of memory for a bigger hash-table, move their data into the larger chunk and free the previous chunk. If those freed chunks don't get reused, then we will not achieve the optimum memory savings. Allowing the sketches to allocate and free memory directly from the operating system allows us to leverage the already existing and very sophisticated malloc() and free() of the underlying C-libraries and the OS itself. On the flip side, the challenge of _not_ using a chunk of the processing buffer is that we need two things to make that work: 1. A way for users to configure the maximum amount of 'extra' memory used per query, or some mechanism to set that limit automatically. (We can't allow unlimited allocation, since we'll risk running out of memory.) 2. A way to track the amount of memory being used per query, and if it exceeds the limit, trigger the out-of-memory code that would get triggered if the processing buffer filled up. Depending on where we are in the query pipeline and what engine is being used, that might trigger a spill to disk, a second pass through the data, an emitting of partially data to a later stage of the pipeline, or an error. They are probably solvable, but I just wanted to call them out. > BTW, there is already a mechanism in the JVM to track usage of off-heap memory used by DirectByteBuffers. We use this same mechanism to track allocations we make against off-heap memory even though we don't use DirectByteBuffers. So the JVM knows and tracks this usage already. Druid could also monitor these same JVM counters if it wanted or needed to. Do you know if there's a way to use this functionality to track per-query memory usage? That would be ideal, since we don't want excessive memory use by one query to affect another unrelated query.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
