jihoonson commented on a change in pull request #12128: URL: https://github.com/apache/druid/pull/12128#discussion_r783499637
########## File path: docs/operations/mixed-workloads.md ########## @@ -0,0 +1,211 @@ +--- +id: mixed-workloads +title: Configure Druid for mixed workloads +sidebar_label: Mixed workloads +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +If you frequently run concurrent, heterogeneous workloads on your Apache Druid cluster, configure Druid to properly allocate cluster resources to optimize your overall query performance. + +Each Druid query consumes a certain amount of cluster resources, such as processing threads, memory buffers for intermediate query results, and HTTP threads for communicating between Brokers and data servers. +"Heavy" queries that return large results are more resource-intensive than short-running, "light" queries. +You typically do not want these long resource-intensive queries to throttle the performance of short interactive queries. +For example, if you run both sets of queries in the same Druid cluster, heavy queries may employ all available HTTP threads. +This situation slows down subsequent queries—heavy and light—and may trigger timeout errors for those queries. +With proper resource isolation, you can execute long-running, low priority queries without interfering with short-running, high priority queries. + +Druid provides the following strategies to isolate resources and improve query concurrency: +- **Query laning** where you set a limit on the maximum number of long-running queries executed on each Broker. +- **Service tiering** which defines separate groups of Historicals and Brokers to receive different query assignments based on query priority. + +You can profile Druid queries using normal performance profiling techniques such as Druid query metrics analysis, thread dumps of JVM, or flame graphs to identify what resources are affected by mixed workloads. +The largest bottleneck will likely be in the Broker HTTP threads. +Mitigate resource contention of the Broker HTTP threads with query laning. +However, mixed workloads also affect other resources, including processing threads and merge buffers. +Reduce the burden on these resources by applying service tiering in addition to query laning. + + +## Query laning + +Query laning directs Druid to restrict resource usage for less urgent queries to ensure dedicated resources for high priority queries. Query laning is ideal when you need to run many concurrent queries having heterogeneous workloads. Review comment: OK. I realized that everyone may read the word "ideal" differently. Query laning is a solution that just works well in many cases with an easy setup, but is not a complete solution because it has a limitation on the resource type today that you can control with it. It means, query laning may not work well in some edge cases. How about "You will want to consider query laning as the first option you would try when you need to run many concurrent queries..."? ########## File path: docs/operations/mixed-workloads.md ########## @@ -0,0 +1,211 @@ +--- +id: mixed-workloads +title: Configure Druid for mixed workloads +sidebar_label: Mixed workloads +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +If you frequently run concurrent, heterogeneous workloads on your Apache Druid cluster, configure Druid to properly allocate cluster resources to optimize your overall query performance. + +Each Druid query consumes a certain amount of cluster resources, such as processing threads, memory buffers for intermediate query results, and HTTP threads for communicating between Brokers and data servers. +"Heavy" queries that return large results are more resource-intensive than short-running, "light" queries. +You typically do not want these long resource-intensive queries to throttle the performance of short interactive queries. +For example, if you run both sets of queries in the same Druid cluster, heavy queries may employ all available HTTP threads. +This situation slows down subsequent queries—heavy and light—and may trigger timeout errors for those queries. +With proper resource isolation, you can execute long-running, low priority queries without interfering with short-running, high priority queries. + +Druid provides the following strategies to isolate resources and improve query concurrency: +- **Query laning** where you set a limit on the maximum number of long-running queries executed on each Broker. +- **Service tiering** which defines separate groups of Historicals and Brokers to receive different query assignments based on query priority. + +You can profile Druid queries using normal performance profiling techniques such as Druid query metrics analysis, thread dumps of JVM, or flame graphs to identify what resources are affected by mixed workloads. +The largest bottleneck will likely be in the Broker HTTP threads. +Mitigate resource contention of the Broker HTTP threads with query laning. +However, mixed workloads also affect other resources, including processing threads and merge buffers. +Reduce the burden on these resources by applying service tiering in addition to query laning. + + +## Query laning + +Query laning directs Druid to restrict resource usage for less urgent queries to ensure dedicated resources for high priority queries. Query laning is ideal when you need to run many concurrent queries having heterogeneous workloads. + +Query lanes are analogous to carpool and normal lanes on the freeway. With query laning, Druid sets apart VIP lanes from other general lanes. +Druid restricts low priority queries to the general lanes and allows high priority queries to run wherever possible, whether in a VIP or general lane. + +In Druid, query lanes reserve resources for Broker HTTP threads. Each Druid query requires one Broker thread. The number of threads on a Broker is defined by the `druid.server.http.numThreads` parameter. Broker threads may be occupied by tasks other than queries, such as health checks. You can use query laning to limit the number of HTTP threads designated for resource-intensive queries, leaving other threads available for short-running queries and other tasks. + +### General properties + +Set the following query laning properties in the `broker/runtime.properties` file. + +* `druid.query.scheduler.laning.strategy` – The strategy used to assign queries to lanes. +You can use the built-in [“high/low” laning strategy](../configuration/index.md#highlow-laning-strategy), or [define your own laning strategy manually](../configuration/index.md#manual-laning-strategy). +* `druid.query.scheduler.numThreads` – The total number of queries that can be served per Broker. We recommend setting this value to 1-2 less than `druid.server.http.numThreads`. + > The query scheduler by default does not limit the number of Broker HTTP threads. Setting this property to a bounded number limits the thread count. If the allocated threads are all occupied, any incoming query, including interactive queries, will be rejected with an HTTP 429 status code. + +### Lane-specific properties + +If you use the __high/low laning strategy__, set the following: + +* `druid.query.scheduler.laning.maxLowPercent` – The maximum percent of query threads to handle low priority queries. The remaining query threads are dedicated to high priority queries. + +Consider also defining a [prioritization strategy](../configuration/index.md#prioritization-strategies) for the Broker to label queries as high or low priority. Otherwise, manually set the priority for incoming queries on the [query context](../querying/query-context.md). + +If you use a __manual laning strategy__, set the following: + +* `druid.query.scheduler.laning.lanes.{name}` – The limit for how many queries can run in the `name` lane. Define as many named lanes as needed. +* `druid.query.scheduler.laning.isLimitPercent` – Whether to treat the lane limit as an exact number or a percent of the minimum of `druid.server.http.numThreads` or `druid.query.scheduler.numThreads`. + +With manual laning, incoming queries can be labeled with the desired lane in the `lane` parameter of the [query context](../querying/query-context.md). + +See [Query prioritization and laning](../configuration/index.md#query-prioritization-and-laning) for additional details on query laning configuration. + +### Example + +Example config for query laning with the high/low laning strategy: + +``` +# Laning strategy +druid.query.scheduler.laning.strategy=hilo +druid.query.scheduler.laning.maxLowPercent=20 + +# Limit the number of HTTP threads for query processing +# This value should be less than druid.server.http.numThreads +druid.query.scheduler.numThreads=40 +``` + + +## Service tiering + +In service tiering, you define separate groups of Historicals and Brokers to manage queries based on the segments and resource requirements of the query. +You can limit the resources that are set aside for certain types of queries. +Many heavy queries involving complex subqueries or large result sets can crash a Broker, or worse, take down a cluster. +Minimize the impact of these heavy queries by limiting them to a separate Broker tier. +When all Brokers set aside for heavy queries are occupied, users must wait to submit additional heavy queries until the designated resources become available. Review comment: This looks a bit unclear to me. I think it's because there is nothing in Druid disallowing users to submit additional queries. Perhaps ```suggestion When all Brokers set aside for heavy queries are occupied, additional queries submitted subsequently can wait until the designated resources become available, which can fail those queries after query timeout expires. ``` ########## File path: docs/operations/mixed-workloads.md ########## @@ -0,0 +1,211 @@ +--- +id: mixed-workloads +title: Configure Druid for mixed workloads +sidebar_label: Mixed workloads +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +If you frequently run concurrent, heterogeneous workloads on your Apache Druid cluster, configure Druid to properly allocate cluster resources to optimize your overall query performance. + +Each Druid query consumes a certain amount of cluster resources, such as processing threads, memory buffers for intermediate query results, and HTTP threads for communicating between Brokers and data servers. +"Heavy" queries that return large results are more resource-intensive than short-running, "light" queries. +You typically do not want these long resource-intensive queries to throttle the performance of short interactive queries. +For example, if you run both sets of queries in the same Druid cluster, heavy queries may employ all available HTTP threads. +This situation slows down subsequent queries—heavy and light—and may trigger timeout errors for those queries. +With proper resource isolation, you can execute long-running, low priority queries without interfering with short-running, high priority queries. + +Druid provides the following strategies to isolate resources and improve query concurrency: +- **Query laning** where you set a limit on the maximum number of long-running queries executed on each Broker. +- **Service tiering** which defines separate groups of Historicals and Brokers to receive different query assignments based on query priority. + +You can profile Druid queries using normal performance profiling techniques such as Druid query metrics analysis, thread dumps of JVM, or flame graphs to identify what resources are affected by mixed workloads. +The largest bottleneck will likely be in the Broker HTTP threads. +Mitigate resource contention of the Broker HTTP threads with query laning. +However, mixed workloads also affect other resources, including processing threads and merge buffers. +Reduce the burden on these resources by applying service tiering in addition to query laning. + + +## Query laning + +Query laning directs Druid to restrict resource usage for less urgent queries to ensure dedicated resources for high priority queries. Query laning is ideal when you need to run many concurrent queries having heterogeneous workloads. + +Query lanes are analogous to carpool and normal lanes on the freeway. With query laning, Druid sets apart VIP lanes from other general lanes. +Druid restricts low priority queries to the general lanes and allows high priority queries to run wherever possible, whether in a VIP or general lane. + +In Druid, query lanes reserve resources for Broker HTTP threads. Each Druid query requires one Broker thread. The number of threads on a Broker is defined by the `druid.server.http.numThreads` parameter. Broker threads may be occupied by tasks other than queries, such as health checks. You can use query laning to limit the number of HTTP threads designated for resource-intensive queries, leaving other threads available for short-running queries and other tasks. + +### General properties + +Set the following query laning properties in the `broker/runtime.properties` file. + +* `druid.query.scheduler.laning.strategy` – The strategy used to assign queries to lanes. +You can use the built-in [“high/low” laning strategy](../configuration/index.md#highlow-laning-strategy), or [define your own laning strategy manually](../configuration/index.md#manual-laning-strategy). +* `druid.query.scheduler.numThreads` – The total number of queries that can be served per Broker. We recommend setting this value to 1-2 less than `druid.server.http.numThreads`. + > The query scheduler by default does not limit the number of Broker HTTP threads. Setting this property to a bounded number limits the thread count. If the allocated threads are all occupied, any incoming query, including interactive queries, will be rejected with an HTTP 429 status code. Review comment: I'm not sure if readers would tell that the doc talks about the same thread in these two lines. Would it be clearer if it says ```suggestion * `druid.query.scheduler.numThreads` – The total number of queries that can be served per Broker. We recommend setting this value to 1-2 less than `druid.server.http.numThreads`. > The query scheduler by default does not limit the number of queries that a Broker can serve. Setting this property to a bounded number limits the thread count. If the allocated threads are all occupied, any incoming query, including interactive queries, will be rejected with an HTTP 429 status code. ``` ########## File path: docs/operations/mixed-workloads.md ########## @@ -0,0 +1,211 @@ +--- +id: mixed-workloads +title: Configure Druid for mixed workloads +sidebar_label: Mixed workloads +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +If you frequently run concurrent, heterogeneous workloads on your Apache Druid cluster, configure Druid to properly allocate cluster resources to optimize your overall query performance. + +Each Druid query consumes a certain amount of cluster resources, such as processing threads, memory buffers for intermediate query results, and HTTP threads for communicating between Brokers and data servers. +"Heavy" queries that return large results are more resource-intensive than short-running, "light" queries. +You typically do not want these long resource-intensive queries to throttle the performance of short interactive queries. +For example, if you run both sets of queries in the same Druid cluster, heavy queries may employ all available HTTP threads. +This situation slows down subsequent queries—heavy and light—and may trigger timeout errors for those queries. +With proper resource isolation, you can execute long-running, low priority queries without interfering with short-running, high priority queries. + +Druid provides the following strategies to isolate resources and improve query concurrency: +- **Query laning** where you set a limit on the maximum number of long-running queries executed on each Broker. +- **Service tiering** which defines separate groups of Historicals and Brokers to receive different query assignments based on query priority. + +You can profile Druid queries using normal performance profiling techniques such as Druid query metrics analysis, thread dumps of JVM, or flame graphs to identify what resources are affected by mixed workloads. +The largest bottleneck will likely be in the Broker HTTP threads. +Mitigate resource contention of the Broker HTTP threads with query laning. +However, mixed workloads also affect other resources, including processing threads and merge buffers. +Reduce the burden on these resources by applying service tiering in addition to query laning. + + +## Query laning + +Query laning directs Druid to restrict resource usage for less urgent queries to ensure dedicated resources for high priority queries. Query laning is ideal when you need to run many concurrent queries having heterogeneous workloads. + +Query lanes are analogous to carpool and normal lanes on the freeway. With query laning, Druid sets apart VIP lanes from other general lanes. +Druid restricts low priority queries to the general lanes and allows high priority queries to run wherever possible, whether in a VIP or general lane. + +In Druid, query lanes reserve resources for Broker HTTP threads. Each Druid query requires one Broker thread. The number of threads on a Broker is defined by the `druid.server.http.numThreads` parameter. Broker threads may be occupied by tasks other than queries, such as health checks. You can use query laning to limit the number of HTTP threads designated for resource-intensive queries, leaving other threads available for short-running queries and other tasks. + +### General properties + +Set the following query laning properties in the `broker/runtime.properties` file. + +* `druid.query.scheduler.laning.strategy` – The strategy used to assign queries to lanes. +You can use the built-in [“high/low” laning strategy](../configuration/index.md#highlow-laning-strategy), or [define your own laning strategy manually](../configuration/index.md#manual-laning-strategy). +* `druid.query.scheduler.numThreads` – The total number of queries that can be served per Broker. We recommend setting this value to 1-2 less than `druid.server.http.numThreads`. + > The query scheduler by default does not limit the number of Broker HTTP threads. Setting this property to a bounded number limits the thread count. If the allocated threads are all occupied, any incoming query, including interactive queries, will be rejected with an HTTP 429 status code. + +### Lane-specific properties + +If you use the __high/low laning strategy__, set the following: + +* `druid.query.scheduler.laning.maxLowPercent` – The maximum percent of query threads to handle low priority queries. The remaining query threads are dedicated to high priority queries. + +Consider also defining a [prioritization strategy](../configuration/index.md#prioritization-strategies) for the Broker to label queries as high or low priority. Otherwise, manually set the priority for incoming queries on the [query context](../querying/query-context.md). + +If you use a __manual laning strategy__, set the following: + +* `druid.query.scheduler.laning.lanes.{name}` – The limit for how many queries can run in the `name` lane. Define as many named lanes as needed. +* `druid.query.scheduler.laning.isLimitPercent` – Whether to treat the lane limit as an exact number or a percent of the minimum of `druid.server.http.numThreads` or `druid.query.scheduler.numThreads`. + +With manual laning, incoming queries can be labeled with the desired lane in the `lane` parameter of the [query context](../querying/query-context.md). + +See [Query prioritization and laning](../configuration/index.md#query-prioritization-and-laning) for additional details on query laning configuration. + +### Example + +Example config for query laning with the high/low laning strategy: + +``` +# Laning strategy +druid.query.scheduler.laning.strategy=hilo +druid.query.scheduler.laning.maxLowPercent=20 + +# Limit the number of HTTP threads for query processing +# This value should be less than druid.server.http.numThreads +druid.query.scheduler.numThreads=40 +``` + + +## Service tiering + +In service tiering, you define separate groups of Historicals and Brokers to manage queries based on the segments and resource requirements of the query. +You can limit the resources that are set aside for certain types of queries. +Many heavy queries involving complex subqueries or large result sets can crash a Broker, or worse, take down a cluster. Review comment: This sounds a bit scarier than necessary as it technically can happen but is not common. Can we mention the problem of resource contention between long-running queries and short-running queries instead? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
