sanastas commented on issue #7676: Add OakIncrementalIndex to Druid URL: https://github.com/apache/incubator-druid/pull/7676#issuecomment-494300561 Hi Slim (@b-slim ), I am impressed with such a thorough review! Thanks! It looks like the issue of lock-free programming and lock-free memory management is as close to your heart as it is close to mine! This is the reason I did PhD in this field :) I am glad to find an ally! What was your PhD thesis about? I have some answers below. If I understand something wrongly, I would be happy to hear and learn. > that is simple not true, some of those locks are part of allocate calls and some are part of iterator loop to do some stuff that i doubt it is static ... I would be happy to speak with you about internal Oak implementation. As it is time consuming to discuss it via chat, I suggest us to arrange us an off-line discussion via Hangouts for example. Let me know what time-slots suit you. > Looking at the allocator briefly i see bunch of red flags, like possible internal fragmentations up to factor 2 and not clear way to deal with external fragmentations … Memory Management is a very hard thing in general and make it lock free is not going to be simple task .... We're very well aware that the full lock-free Memory Management is a hard thing and actually not available yet. However, the Druid's usage of IncrementalIndex is a restricted one, IMHO. For example, there are no concurrent deletions, thus dealing with internal fragmentation should not be an issue. Dealing with external fragmentation might be an issue of tuning per specific Block size and common workloads. Bottom line, we had no intent to write a perfect general-purpose full lock-free Memory Management, but a specialized memory management. Not to say that it is not final and can be fixed/replaced if some objective issues are found. > Writing one form scratch is a common source of bugs, In fact most of the big data projects avoid this task and use the Netty allocator. As you have seen our memory manager and memory allocator is quite restricted. So we believe it can perform faster and it can be easier to find bugs there. Thanks for referring to the Netty Allocator. I have taken a brief look on Netty Allocator, it appears to be based on reference-counting and their underlying data structure is their own ByteBuf (not ByteBuffer). It is assumed that epoch-based GC is more efficient in performance than reference-counting. But mostly important, I am not sure we can convert ByteBuf into ByteBuffer without any complications or performance issues. Nor can we require Druid to start working with Netty ByteBufs. However, we will continue investigation and if Netty Allocator can be used and gives a better performance, no problems to switch to it. > The take our of this, current state is not lock free thus lets avoid confusion, (FYI even on the oak github page i do not see it called lock-free so not sure why you are calling it lock free). Generally speaking lock-free is about the data structure not about the underlying memory management. As a full lock-free memory management (GC+allocation) isn’t available yet, not any of currently existing lock-free data structures can be called lock-free and neither their performance results are valid… Huge amount of scientific papers wasted :) . But I will remove the "lock-free” term from anywhere it appears here. As I said, I am a big fan of lock-free programming, and I will be the first one to fight to eliminate any lock anywhere. But in reality what does matter is the final performance results and in order to see gains from lock-free programming we need big contention of multi-threading, is it the case in Druid’s IncrementalIndex? All the benchmarks are single threaded… I am OK with lock not on the hot path and with lock that doesn't impact the performance at all. > But the most important thing to add here is how OKA memory management will fit with how Druid manages Direct memory. Exactly! This is the mostly important! And here we need your input! How does Druid manages Direct memory? Is there some policy? Documentation to read? Pointers to code?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
