rbankar7 commented on issue #19937: URL: https://github.com/apache/druid/issues/19937#issuecomment-5227001354
Thanks @FrankChen021. >I'm wondering do you have many "small" segments in your historical nodes? and what is the QPS on your historical nodes? Yes — on this cluster we create segments every minute, historical QPS is ~4–5k, and it's almost entirely 1-min and 3-min-interval groupBy queries. Because the segments are small and per-minute, each query fans out over many segments, so the per-segment task rate through the processing pool's single queue is very high — that's the crux: we were running numThreads = num_cores − 1 (55 here), but the single queue lock, not CPU, was the ceiling, so we couldn't actually use the available cores. Scaling out horizontally does relieve it, but it increases query fan-out and therefore merge overhead on the brokers — so it trades a historical-side bottleneck for a broker-side one, and there's a practical limit to how far we can push it. That's the motivation for this proposal. On the semantics — you flagged the trade-off accurately. To keep it safe: Default numThreadPools=1, which is identical to today's behavior (one queue, global priority + global FIFO). Nothing changes unless an operator opts in. We can document the trade-off explicitly on the config, and call out exactly when it helps — i.e. when numThreads and/or the number of segments scanned per node is high enough that the single queue lock becomes the bottleneck. It's also worth noting that priority/FIFO ordering in the processing pool is already a per-process property, not cluster-global — a query fans out to many data servers, and each historical's pool orders independently, so there is no global ordering across the fleet today. The number of independent ordering domains already grows every time we scale out . NumThreadPools applies the same idea inside a process, so it lets us relieve this contention without adding nodes and the broker-side fan-out cost that comes with them. (To be precise about the one difference: scaling out keeps each node's single queue globally ordered, just with fewer tasks, whereas numThreadPools relaxes ordering within a process. So it isn't strictly identical — but since ordering is already only process-local, and both approaches simply make each independent queue smaller, this feels like a modest, opt-in step in a direction operators already rely on.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
