Eshcar commented on issue #263: Theta sketch - Concurrent union implementation URL: https://github.com/apache/incubator-datasketches-java/issues/263#issuecomment-509524373 I get the message. Locks are a good enough solution for incremental index since 1) ingestion is done by a single thread, and vast majority of sketches are small to medium size, 2) reads arrive at a low rate, and queries rarely need to read or aggregate large sketches. Fair enough :) A question that is not related to incremental indexing. I assume aggregation at query time are executed in memory, even when data already resides on disk. What I suggest here is to consider a way to expedite query processing by having multiple threads (can be a small number) executing the aggregation in parallel, instead of sequentially by a single thread. For example if a query needs to aggregate (union) thousands of sketches then perhaps parallel processing is better. I believe this can reduce query latency, and specifically significantly reduce high percentiles query latency. It should be straightforward to understand if a query needs to aggregate a large number of sketches and only for these cases trigger parallel union (by using concurrent union). Does this seems like something that can improve Druid performance?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
