jihoonson commented on issue #7838: Improve IncrementalIndex concurrency scalability URL: https://github.com/apache/incubator-druid/pull/7838#issuecomment-504597636 > Start using the Oak-based index with more writer threads, to let it really shine (the microbenchmarks show that it becomes more attractive as the number of writers scales). This might require more far-fetched changes, which we might not fully appreciate at the moment, so let's discuss. @ebortnik thanks! Your comment makes things more clear. I have a follow-up question on the above comment though. As I mentioned earlier, our previous work has focused on distributed data ingestion in which multiple tasks can run at the same time to ingest data into the same dataSource. Each task uses a few threads for indexing (basically read thread, persist thread and merge thread), but they can run in parallel in multiple middleManagers (or even same middleManager) so that faster ingestion could be achieved. It seems that your comment assumes that multi-thread indexing will bring us other benefits which is missing in distributed data ingestion. What kind of benefits can we expect with multi-thread indexing? And how is it different from what we can get from distributed indexing?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
