Eshcar commented on issue #263: Theta sketch - Concurrent union implementation URL: https://github.com/apache/incubator-datasketches-java/issues/263#issuecomment-508495971 Thanks @himanshug, @gianm, I am aware of the work done by Anastasia and Eran on integrating Oak into Druid, and follow the Issues and PRs very closely :) Let me elaborate some more on our implementation and benchmarks results. **Ingestion time aggregators: single-writer-multiple-readers** From the Oak-related discussions and also from our previous conversation on the Druid dev list I understand the real-time indexer access is single-writer-multiple-reader. Concurrent Union (and in general concurrent sketch) is also beneficial for this use case, specifically when handling big aggregations. We've run benchmarks that demonstrated the benefits of single thread concurrent theta sketch over lock-based implementation, when sketches are big - I can share the results, or better I should publish them in the data sketch web site. We mainly measured write throughput, since it is hard to measure read latency percentiles with high accuracy. It is likely to assume that the latency of readers improve as they are simply reading an attribute instead of taking a lock and accessing more than one attribute. I believe we will reproduce these results for concurrent Union. > that sounds like there are extra threads doing things in the background. I wonder if that and needs for creating snapshot offsets(or worsens) the performance improvements received by removing the lock in the context of Druid. You are right there is a background thread. For small sketches we have a workaround where the updating thread itself does the propagation. Creating a snapshot for theta sketch is very simple and efficient. When considering single writer thread in small sketches indeed the overhead of managing local buffers and shared sketch offsets the benefit of removing the lock, but from a certain size (number of uniques) the improvement outweighs the overhead. **Query time aggregators** This is the case where there are no updates concurrent to queries (right?). So the current implementation does not use locks. Our evaluation results show that when the sketches are VERY big (more than 1M uniques) using multiple threads can double and even triple the updating rate (vs single thread with no lock). So if there is a way to identify potential heavy aggregation queries, considering running multiple threads of the concurrent implementation can result in a significant reduction of query latency. We never tested sketches in the context of Druid. Once we have the implementation of the concurrent Union we can start playing with Druid queries. Back to my original question, I think I will implement design option 1, it means implementing one additional class, and have many method throwing exception, but it is the most reasonable option in order to be compatible with application which use the existing union implementation. Long answer :) Please let me know if there are more comments or questions. And thanks again for the feedback.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
