clintropolis commented on a change in pull request #9407: query laning and load shedding URL: https://github.com/apache/druid/pull/9407#discussion_r390198573
########## File path: docs/configuration/index.md ########## @@ -1481,9 +1481,35 @@ These Broker configurations can be defined in the `broker/runtime.properties` fi |`druid.broker.select.tier`|`highestPriority`, `lowestPriority`, `custom`|If segments are cross-replicated across tiers in a cluster, you can tell the broker to prefer to select segments in a tier with a certain priority.|`highestPriority`| |`druid.broker.select.tier.custom.priorities`|`An array of integer priorities.`|Select servers in tiers with a custom priority list.|None| +##### Query laning + +*Laning strategies* allow you to control capacity utilization for heterogeneous query workloads. With laning, the broker examines and classifies a query for the purpose of assigning it to a 'lane'. Lanes have capacity limits, enforced by the broker, that can be used to ensure sufficient resources are available for other lanes or for interactive queries (with no lane), or to limit overall throughput for queries within the lane. Requests in excess of the capacity are discarded with an HTTP 429 status code. + +|Property|Description|Default| +|--------|-----------|-------| +|`druid.query.scheduler.numThreads`|Maximum number of HTTP threads to dedicate to query processing. To save HTTP thread capacity, this should be lower than `druid.server.http.numThreads`.|Unbounded| Review comment: >in what use case would I ever want to set it something other than druid.server.http.numThreads ? I actually think we might always want to set it lower than `druid.server.http.numThreads`, but I was too nervous to make this the default and made it opt in behavior instead (since it grabs and releases locks for each query if there is some bug in releasing locks a broker would eventually stop accepting queries entirely). The primary reason I think we want it lower than `druid.server.http.numThreads` is to save some 'slack' space for non-query http connections, like accepting health checks, lookup management, and other such things that can be starved when long running queries start to pile up. >If the recommendation is to set it lower than `druid.server.http.numThreads` then why the default value is not set to `druid.server.http.numThreads - 1` ? See my above nervousness, but I think `druid.server.http.numThreads - 1` would probably be a good default. This might want to be adjusted to be even lower depending on how much other non query http traffic the server is receiving (e.g. frequently polled/updated lookups, etc). >I guess, as a user I don't quite understand the importance of setting this higher/same/lower compared to druid.server.http.numThreads and when I should choose one vs the other. I would agree the current documentation doesn't quite adequately describe how this stuff might be utilized, in a future PR i want to add a section to [cluster tuning docs](https://github.com/apache/druid/blob/master/docs/operations/basic-cluster-tuning.md) to more properly advise on when and how to set this stuff up. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
