eranmeir edited a comment on issue #7838: Improve IncrementalIndex concurrency scalability URL: https://github.com/apache/incubator-druid/pull/7838#issuecomment-502591292 @leventov Thanks for the review. I will soon add a commit that addresses the issues you raise. I have updated the attached pdf with raw numbers of pre- and post-fix benchmarks. @jihoonson Thanks for the feedback. We used Oak for indexing large amounts of data and saw significant performance gains from off-heap storage and multi-threaded indexing. This separate PR may be thought of as ground work for Oak, but multi-threaded performance gains are obviously independent. The way I understand what you write, queries that happen before segment merge may benefit from increase in indexing throughput. We haven't tried comparing costs of index building and segment merging. Perhaps we can try and leverage multi-threading in the merge process. I'll be happy to discuss this.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
