LakshSingla commented on PR #14808: URL: https://github.com/apache/druid/pull/14808#issuecomment-1695071634
@abhishekagarwal87 Currently there is a limit per query in the broker guardrail. In the case of the example, let's say that overall 1/6th of broker memory is allocated for all the subqueries. The code would divide the 1/6th portion to the maximum number of queries that can be run on the system (limited to the number of broker threads), and each subquery has a local counter for the size. With the global counter, I see that while it would allow more subqueries to pass in the happy case when no other is running, this can also potentially mean: 1. There would be resource contention amongst the various queries that are running on the system. 2. Oncoming queries can fail if the existing subqueries are hogging up the space, even if they don't require much space to materialize the results 3. In the worst case, subquery A can materialize some rows, subquery B can materialize some rows, and both fill up the shared space and then realize that none of them have enough space to continue, causing both queries to return resource limit exceeded. While the above cases are slightly hyperbolized, we still would need to think about them before maintaining a global counter. Therefore "isolating" each query's available heap space seems to be better for the time moment, till we tackle the above. (FWIW, is also what the behavior with `maxSubqueryRows` is) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
