Nagarjunanm commented on issue #20494: URL: https://github.com/apache/druid/issues/20494#issuecomment-6264882489
Hi @gianm, I'd like to pick this up, if nobody else is working on it. I reproduced it with a test that uses the Broker's real buffer pools. With 1 MB buffers, 4 merge buffers, and direct memory set to exactly what the startup check requires (5 MB), 6 concurrent groupby's on an inline datasource allocated 4 processing buffers, so 8 MB were in use. I'm planning to go with option 1: - Add a Broker setting, `druid.processing.local.numThreads`, defaulting to 1. Local queries would run their per-segment work on a `PrioritizedExecutorService` with that many threads instead of in HTTP threads, so at most that many processing buffers are in use at once. - Change the Broker's direct memory check and the docs to `sizeBytes * (numMergeBuffers + numThreads)`. With the default of 1, that's the current formula, so no Broker fails its startup check after upgrading. - Keep `query/cpu/time` accurate by adding the CPU time of the pool threads. With 1 thread, concurrent local groupBy, timeseries, and topN queries would be processed one at a time. Is that default OK with you, or would you rather size the pool another way? And does option 1 work for you, or would you rather go with option 2? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
