This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 9453bd629ea add overview en docs (#1643)
9453bd629ea is described below
commit 9453bd629ea7eb81d7cb367706bf0d975d528b42
Author: yiguolei <[email protected]>
AuthorDate: Fri Dec 27 17:48:03 2024 +0800
add overview en docs (#1643)
## Versions
- [x] dev
- [x] 3.0
- [x] 2.1
- [ ] 2.0
## Languages
- [ ] Chinese
- [x] English
## Docs Checklist
- [ ] Checked by AI
- [ ] Test Cases Built
---------
Co-authored-by: yiguolei <[email protected]>
---
.../workload-management-summary.md | 50 ++++++------
.../workload-management-summary.md | 4 +-
.../workload-management/resource-group.md | 3 +-
.../workload-management-summary.md | 4 +-
.../workload-management/resource-group.md | 2 +-
.../workload-management-summary.md | 8 +-
.../workload-management-summary.md | 86 ++++++++-------------
.../workload-management-summary.md | 89 +++++++++-------------
8 files changed, 105 insertions(+), 141 deletions(-)
diff --git
a/docs/admin-manual/workload-management/workload-management-summary.md
b/docs/admin-manual/workload-management/workload-management-summary.md
index 82f4f040ca2..e5ef94b3f7b 100644
--- a/docs/admin-manual/workload-management/workload-management-summary.md
+++ b/docs/admin-manual/workload-management/workload-management-summary.md
@@ -24,32 +24,38 @@ specific language governing permissions and limitations
under the License.
-->
-The proper allocation of a database's system resources (such as CPU, memory,
I/O, and storage) is crucial. By precisely managing these resources, it is
ensured that queries receive adequate resource support during execution,
thereby enhancing the overall performance and maintaining the stability of the
system. Doris provides resource control functionality, allowing users to divide
the physical resources within the cluster into multiple resource units. This
resource allocation control pre [...]
+Workload management is an very important feature of Doris, playing a critical
role in the overall operation of the system. Through reasonable workload
management strategies, resource utilization can be optimized, system stability
enhanced, and response time reduced. It has the following abilities:
-## Scenarios for Resource Control
-Doris's resource control feature provides refined resource allocation and
management support for concurrent multi-user and multi-business scenarios,
ensuring efficient and stable system operation in complex environments.
Resource control is typically used in the following scenarios:
-- When different business data is stored in a single Doris cluster, resource
control prevents resource contention between different businesses.
-- When different users access the same data, different tenants are isolated to
avoid resource contention among users.
-- In scenarios with multiple concurrent workloads, resource queues adjust
query priorities to prevent faster queries from being blocked by longer-running
ones.
-- Implementing load circuit breakers that automatically terminate workloads
when circuit breaker conditions are met.
+- Resource Isolation: By dividing into multiple groups and setting resource
(CPU, Memory, IO) limits for each group, it ensures that there is no
interference between multiple users or different tasks (such as read and write
operations) of the same user.
-## Methods for Resource Group Division
-In Doris, users can divide system resources into different resource pools
based on specific business needs, ensuring resource independence and mutual
non-interference between pools. By binding resource pools to different users,
Doris achieves refined resource management and control among users. Doris
offers three primary methods for resource division:
-- Resource Group: Uses BE nodes as the smallest granularity and divides
resources into multiple groups by setting tags.
-- Workload Group: Divides resources into multiple groups with CPU/Memory/IO as
the smallest granularity, enabling more detailed resource allocatio
-- Compute Group: A method for dividing resources in a storage-compute
separation model, similar to Resource Group, using BE nodes as the smallest
granularity.
+- Concurrency Control and Queuing: It can limit the number of tasks that can
be executed simultaneously in the entire cluster. When threshold is exceeded,
tasks will be queued.
+- Circuit Breaker: During query planning phase or execution phase, tasks can
be automatically cancelled based on conditions such as the estimated number of
partitions to be read, the amount of data to be scanned, the allocated memory
size, and the execution time. This avoids unreasonable tasks from occupying too
many system resources.
-| Resource Group Division | Granularity of Isolation | Description
of Isolation Solution
| Soft/Hard Limit |
Cross-Group Queries|
-| ---------- | -----------
|---------------------------------------------------------------------------------------------------------|-----|-----|
-| Resource Group | Server node level, fully isolated resources | Sets
tags for BE nodes, with BE nodes sharing the same tag forming a resource group,
enabling node-level resource isolation within the cluster and resource
restriction per query. Facilitates fault isolation, where issues within one
resource group do not affect others. | Hard Limit |Not supported. Must
ensure at least one data replica within the resource group. |
-| Workload Group | Resource granularity, including CPU, memory, IO, etc.
| Utilizes cgroup for CPU, memory, IO, and concurrency resource control;
provides query queuing to avoid resource contention.
| Supports both hard and soft limits | Supported
|
-|Compute Group | Server node level, fully isolated resources |
Based on a storage-compute separation architecture, data is visible across
multiple Compute Groups; enables node-level resource isolation within the
cluster and resource restriction per query without relying on replica tags. |
Hard Limit | Not supported. Must ensure at least one data replica within
the resource group. |
-## Soft and Hard Resource Limits
-In Doris, soft limits (soft limit) and hard limits (hard limit) can be used to
restrict resource usage:
-- Hard Limit: Represents the absolute upper limit of resources that can be
used, which cannot be exceeded by tenants. Once the hard limit is reached,
requests for additional resources will be rejected. Hard limits are generally
used to prevent resource exhaustion within the cluster or contention between
different businesses, ensuring cluster stability and performance.
-- Soft Limit: Represents a recommended upper limit that can be exceeded. When
the system is not busy, tenants can borrow resources from other resource groups
if they exceed the soft limit. However, during peak times or when resources are
contended, tenants exceeding the soft limit will not receive additional
resources.
+## Resource Isolation Methods
-Resource groups divided using Resource Group/Compute Group methods only
support hard limits. Workload Groups support both soft and hard limits. Soft
limits for Workload resources are typically used for managing sudden resource
spikes, such as temporary query peaks or brief data ingestion increases. Hard
limits are used to prevent resource abuse, enabling absolute resource isolation
between different businesses to avoid contention. Reasonable use of soft and
hard limits can prevent cluste [...]
+Doris can divide resource in the following three ways:
+- Resource Group: Divide multiple BE processes into groups setting tag for
each BE.
+
+- Workload Group: The resource (CPU, Memory, IO) within a BE are divided into
multiple resource groups through Cgroup, enabling more fine-grained resource
isolation.
+
+- Compute Group: It is a way of resource partitioning method in
compute-storage decoupled mode. Similar to Resource Group, it also takes BE as
the minimum granularity to divide multiple groups.
+
+The following table records the characteristics and advantageous scenarios of
different resource partitioning methods:
+
+| Resource Isolation Method | Isolation Granularity | Soft/Hard
Limit | Cross Resource Group Query |
+| ---------- | ----------- |-----|-----|
+| Resource Group | BE node level, with complete resource isolation, can
isolate BE failures | Hard limit |Not support. And it is necessary to
ensure that at least one copy of data is stored within the resource group. |
+| Workload Group | Isolation within BE process; cannot isolate BE failures
| Both hard and soft limit
| Support |
+|Compute Group | BE node level, with complete resource isolation,
can isolate BE failures | Hard limit | Not support |
+
+## Soft Limit and Hard Limit
+
+
+- Hard Limit: The hard limit refers to the absolute upper limit of resource
usage that tenants cannot exceed. Once the hard limit is reached, resource
requests for the excess part will be rejected. Hard limit are generally used to
prevent the depletion of resources within the cluster or resource preemption
between different businesses, ensuring the stability and performance of the
cluster.
+
+- Soft Limit: The soft limit is a resource limit that can be exceeded, usually
representing the recommended upper limit of resource usage. When the system is
not busy, if a tenant requests more resources than the soft limit, it can
borrow resources from other groups. When the system is busy and there is
resource contention, if a tenant requests resources exceeding the soft limit,
it will not be able to obtain additional resources.
+
+When using the Resource Group / Compute Group method to partition resources,
only the hard limit mode is supported. When using the Workload Group method to
partition resources, both the soft limit and hard limit of Workload Group are
supported. The soft limit of Workload Group is usually used for sudden resource
control, such as temporary query peaks or short-term increases in data writing.
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/workload-management/workload-management-summary.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/workload-management/workload-management-summary.md
index 7c6840108ac..97dacb28cf8 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/workload-management/workload-management-summary.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/workload-management/workload-management-summary.md
@@ -42,9 +42,9 @@ Doris 可以通过以下3种方式将资源分组：
- Compute Group: 是存算分离模式下的一种资源组划分的方式，与 Resource Group 类似，它也是以 BE
节点为最小粒度，划分出多个资源组。
-下表中记录了不同资源组划分方式的特点及优势场景：
+下表中记录了不同资源划分方式的特点及优势场景：
-| 资源组划分 | 隔离粒度
| 软/硬限制 | 跨资源组查询 |
+| 资源隔离方式 | 隔离粒度
| 软/硬限制 | 跨资源组查询 |
| ---------- | ----------- |-----|-----|
| Resource Group | 服务器节点级别，资源完全隔离；可以隔离BE故障 | 硬限制
|不支持跨资源组查询，必须保证资源组内至少存储一副本数据。 |
| Workload Group | BE 进程内隔离；不能隔离BE故障
| 支持硬限制与软限制 | 支持跨资源组查询 |
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/workload-management/resource-group.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/workload-management/resource-group.md
index ae321359987..b3190ee42b4 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/workload-management/resource-group.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/workload-management/resource-group.md
@@ -24,13 +24,12 @@ specific language governing permissions and limitations
under the License.
-->
-
Resource Group 是存算一体架构下，实现不同的负载之间物理隔离的一种机制，它的基本原理如下图所示：
- 通过Tag的方式，把BE 划分为不同的组，每个组通过tag的名字来标识，比如上图中把host1,host2,host3 都设置为group a,
把host4,host5 都设置为group b；
-- 将表的不同的副本放到不同的分组中，比如上图中table1 所有的副本都位于group a 中， table2的一个副本位于group
a中，一个副本位于group b 中；
+- 将表的不同的副本放到不同的分组中，比如上图中table1 有3个副本，都位于group a 中， table2 有4个副本，其中2个位于group
a中，2个副本位于group b 中；
- 在查询时，根据不同的用户，使用不同的Resource Group，比如online 用户，只能访问host1,host2,host3
上的数据，所以他可以访问table1和table2；但是offline 用户只能访问host4，host5，所以只能访问table2的数据，由于table1
在group b 上没有对应的副本，所以访问会出错。
Resource Group本质上是一种Table副本的放置策略，所以它有以下优势和限制：
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/workload-management/workload-management-summary.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/workload-management/workload-management-summary.md
index c0a5e0ce15a..03091dc1071 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/workload-management/workload-management-summary.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/admin-manual/workload-management/workload-management-summary.md
@@ -30,7 +30,7 @@ under the License.
- 并发控制与排队: 可以限制整个集群同时执行的任务数量，当超过设置的阈值时自动排队；
-- 熔断: 对于执行中的任务，可以根据扫描的数据量，分配的内存大小，执行时间等条件，自动取消任务，避免不合理的任务占用太多的系统资源。
+- 熔断:
在查询的规划阶段或者执行过程中，可以根据预估的或者执行中需要读取的分区数量，扫描的数据量，分配的内存大小，执行时间等条件，自动取消任务，避免不合理的任务占用太多的系统资源。
## 资源划分方式
@@ -42,7 +42,7 @@ Doris 可以通过以下2种方式将资源分组：
下表中记录了不同资源组划分方式的特点及优势场景：
-| 资源组划分 | 隔离粒度
| 软/硬限制 | 跨资源组查询 |
+| 资源隔离方式 | 隔离粒度
| 软/硬限制 | 跨组查询 |
| ---------- | ----------- |-----|-----|
| Resource Group | 服务器节点级别，资源完全隔离；可以隔离BE故障 | 硬限制
|不支持跨资源组查询，必须保证资源组内至少存储一副本数据。 |
| Workload Group | BE 进程内隔离；不能隔离BE故障
| 支持硬限制与软限制 | 支持跨资源组查询 |
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/workload-management/resource-group.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/workload-management/resource-group.md
index b9f8c06f7f7..b3190ee42b4 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/workload-management/resource-group.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/workload-management/resource-group.md
@@ -29,7 +29,7 @@ Resource Group 是存算一体架构下，实现不同的负载之间物理隔
- 通过Tag的方式，把BE 划分为不同的组，每个组通过tag的名字来标识，比如上图中把host1,host2,host3 都设置为group a,
把host4,host5 都设置为group b；
-- 将表的不同的副本放到不同的分组中，比如上图中table1 所有的副本都位于group a 中， table2的一个副本位于group
a中，一个副本位于group b 中；
+- 将表的不同的副本放到不同的分组中，比如上图中table1 有3个副本，都位于group a 中， table2 有4个副本，其中2个位于group
a中，2个副本位于group b 中；
- 在查询时，根据不同的用户，使用不同的Resource Group，比如online 用户，只能访问host1,host2,host3
上的数据，所以他可以访问table1和table2；但是offline 用户只能访问host4，host5，所以只能访问table2的数据，由于table1
在group b 上没有对应的副本，所以访问会出错。
Resource Group本质上是一种Table副本的放置策略，所以它有以下优势和限制：
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/workload-management/workload-management-summary.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/workload-management/workload-management-summary.md
index 76e90f362da..97dacb28cf8 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/workload-management/workload-management-summary.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.0/admin-manual/workload-management/workload-management-summary.md
@@ -26,11 +26,11 @@ under the License.
负载管理是Doris一项非常重要的功能，在整个系统运行中起着非常重要的作用。通过合理的负载管理策略，可以优化资源使用，提高系统的稳定性，降低响应时间。它具备以下功能：
-- 负载隔离: 通过划分多个Group，并且为每个Group都设置一定的资源（CPU, Memory,
IO）限制，确保多个用户之间、同一用户不同的任务（读写操作）之间互不干扰；
+- 资源隔离: 通过划分多个Group，并且为每个Group都设置一定的资源（CPU, Memory,
IO）限制，确保多个用户之间、同一用户不同的任务（例如读写操作）之间互不干扰；
- 并发控制与排队: 可以限制整个集群同时执行的任务数量，当超过设置的阈值时自动排队；
-- 熔断: 对于执行中的任务，可以根据扫描的数据量，分配的内存大小，执行时间等条件，自动取消任务，避免不合理的任务占用太多的系统资源。
+- 熔断:
在查询的规划阶段或者执行过程中，可以根据预估的或者执行中需要读取的分区数量，扫描的数据量，分配的内存大小，执行时间等条件，自动取消任务，避免不合理的任务占用太多的系统资源。
## 资源划分方式
@@ -42,9 +42,9 @@ Doris 可以通过以下3种方式将资源分组：
- Compute Group: 是存算分离模式下的一种资源组划分的方式，与 Resource Group 类似，它也是以 BE
节点为最小粒度，划分出多个资源组。
-下表中记录了不同资源组划分方式的特点及优势场景：
+下表中记录了不同资源划分方式的特点及优势场景：
-| 资源组划分 | 隔离粒度
| 软/硬限制 | 跨资源组查询 |
+| 资源隔离方式 | 隔离粒度
| 软/硬限制 | 跨资源组查询 |
| ---------- | ----------- |-----|-----|
| Resource Group | 服务器节点级别，资源完全隔离；可以隔离BE故障 | 硬限制
|不支持跨资源组查询，必须保证资源组内至少存储一副本数据。 |
| Workload Group | BE 进程内隔离；不能隔离BE故障
| 支持硬限制与软限制 | 支持跨资源组查询 |
diff --git
a/versioned_docs/version-2.1/admin-manual/workload-management/workload-management-summary.md
b/versioned_docs/version-2.1/admin-manual/workload-management/workload-management-summary.md
index 82f4f040ca2..10244c3d69f 100644
---
a/versioned_docs/version-2.1/admin-manual/workload-management/workload-management-summary.md
+++
b/versioned_docs/version-2.1/admin-manual/workload-management/workload-management-summary.md
@@ -1,55 +1,33 @@
----
-{
-"title": "Workload Management Overview",
-"language": "en"
-}
----
-
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-The proper allocation of a database's system resources (such as CPU, memory,
I/O, and storage) is crucial. By precisely managing these resources, it is
ensured that queries receive adequate resource support during execution,
thereby enhancing the overall performance and maintaining the stability of the
system. Doris provides resource control functionality, allowing users to divide
the physical resources within the cluster into multiple resource units. This
resource allocation control pre [...]
-
-## Scenarios for Resource Control
-Doris's resource control feature provides refined resource allocation and
management support for concurrent multi-user and multi-business scenarios,
ensuring efficient and stable system operation in complex environments.
Resource control is typically used in the following scenarios:
-- When different business data is stored in a single Doris cluster, resource
control prevents resource contention between different businesses.
-- When different users access the same data, different tenants are isolated to
avoid resource contention among users.
-- In scenarios with multiple concurrent workloads, resource queues adjust
query priorities to prevent faster queries from being blocked by longer-running
ones.
-- Implementing load circuit breakers that automatically terminate workloads
when circuit breaker conditions are met.
-
-## Methods for Resource Group Division
-In Doris, users can divide system resources into different resource pools
based on specific business needs, ensuring resource independence and mutual
non-interference between pools. By binding resource pools to different users,
Doris achieves refined resource management and control among users. Doris
offers three primary methods for resource division:
-- Resource Group: Uses BE nodes as the smallest granularity and divides
resources into multiple groups by setting tags.
-- Workload Group: Divides resources into multiple groups with CPU/Memory/IO as
the smallest granularity, enabling more detailed resource allocatio
-- Compute Group: A method for dividing resources in a storage-compute
separation model, similar to Resource Group, using BE nodes as the smallest
granularity.
-
-
-| Resource Group Division | Granularity of Isolation | Description
of Isolation Solution
| Soft/Hard Limit |
Cross-Group Queries|
-| ---------- | -----------
|---------------------------------------------------------------------------------------------------------|-----|-----|
-| Resource Group | Server node level, fully isolated resources | Sets
tags for BE nodes, with BE nodes sharing the same tag forming a resource group,
enabling node-level resource isolation within the cluster and resource
restriction per query. Facilitates fault isolation, where issues within one
resource group do not affect others. | Hard Limit |Not supported. Must
ensure at least one data replica within the resource group. |
-| Workload Group | Resource granularity, including CPU, memory, IO, etc.
| Utilizes cgroup for CPU, memory, IO, and concurrency resource control;
provides query queuing to avoid resource contention.
| Supports both hard and soft limits | Supported
|
-|Compute Group | Server node level, fully isolated resources |
Based on a storage-compute separation architecture, data is visible across
multiple Compute Groups; enables node-level resource isolation within the
cluster and resource restriction per query without relying on replica tags. |
Hard Limit | Not supported. Must ensure at least one data replica within
the resource group. |
-
-## Soft and Hard Resource Limits
-In Doris, soft limits (soft limit) and hard limits (hard limit) can be used to
restrict resource usage:
-- Hard Limit: Represents the absolute upper limit of resources that can be
used, which cannot be exceeded by tenants. Once the hard limit is reached,
requests for additional resources will be rejected. Hard limits are generally
used to prevent resource exhaustion within the cluster or contention between
different businesses, ensuring cluster stability and performance.
-- Soft Limit: Represents a recommended upper limit that can be exceeded. When
the system is not busy, tenants can borrow resources from other resource groups
if they exceed the soft limit. However, during peak times or when resources are
contended, tenants exceeding the soft limit will not receive additional
resources.
-
-Resource groups divided using Resource Group/Compute Group methods only
support hard limits. Workload Groups support both soft and hard limits. Soft
limits for Workload resources are typically used for managing sudden resource
spikes, such as temporary query peaks or brief data ingestion increases. Hard
limits are used to prevent resource abuse, enabling absolute resource isolation
between different businesses to avoid contention. Reasonable use of soft and
hard limits can prevent cluste [...]
+Workload management is an very important feature of Doris, playing a critical
role in the overall operation of the system. Through reasonable workload
management strategies, resource utilization can be optimized, system stability
enhanced, and response time reduced. It has the following abilities:
+
+- Resource Isolation: By dividing into multiple groups and setting resource
(CPU, Memory, IO) limits for each group, it ensures that there is no
interference between multiple users or different tasks (such as read and write
operations) of the same user.
+
+- Concurrency Control and Queuing: It can limit the number of tasks that can
be executed simultaneously in the entire cluster. When threshold is exceeded,
tasks will be queued.
+
+- Circuit Breaker: During query planning phase or execution phase, tasks can
be automatically cancelled based on conditions such as the estimated number of
partitions to be read, the amount of data to be scanned, the allocated memory
size, and the execution time. This avoids unreasonable tasks from occupying too
many system resources.
+
+
+## Resource Isolation Methods
+
+Doris can divide resource in the following three ways:
+
+- Resource Group: Divide multiple BE processes into groups setting tag for
each BE.
+
+- Workload Group: The resource (CPU, Memory, IO) within a BE are divided into
multiple resource groups through Cgroup, enabling more fine-grained resource
isolation.
+
+The following table records the characteristics and advantageous scenarios of
different resource partitioning methods:
+
+| Resource Isolation Method | Isolation Granularity | Soft/Hard
Limit | Cross Resource Group Query |
+| ---------- | ----------- |-----|-----|
+| Resource Group | BE node level, with complete resource isolation, can
isolate BE failures | Hard limit |Not support. And it is necessary to
ensure that at least one copy of data is stored within the resource group. |
+| Workload Group | Isolation within BE process; cannot isolate BE failures
| Both hard and soft limit
| Support |
+
+## Soft Limit and Hard Limit
+
+
+- Hard Limit: The hard limit refers to the absolute upper limit of resource
usage that tenants cannot exceed. Once the hard limit is reached, resource
requests for the excess part will be rejected. Hard limit are generally used to
prevent the depletion of resources within the cluster or resource preemption
between different businesses, ensuring the stability and performance of the
cluster.
+
+- Soft Limit: The soft limit is a resource limit that can be exceeded, usually
representing the recommended upper limit of resource usage. When the system is
not busy, if a tenant requests more resources than the soft limit, it can
borrow resources from other groups. When the system is busy and there is
resource contention, if a tenant requests resources exceeding the soft limit,
it will not be able to obtain additional resources.
+
+When using Resource Group method to partition resources, only hard limit mode
is supported. When using the Workload Group method to partition resources, both
the soft limit and hard limit of Workload Group are supported. The soft limit
of Workload Group is usually used for sudden resource control, such as
temporary query peaks or short-term increases in data writing.
diff --git
a/versioned_docs/version-3.0/admin-manual/workload-management/workload-management-summary.md
b/versioned_docs/version-3.0/admin-manual/workload-management/workload-management-summary.md
index 82f4f040ca2..8432e653e23 100644
---
a/versioned_docs/version-3.0/admin-manual/workload-management/workload-management-summary.md
+++
b/versioned_docs/version-3.0/admin-manual/workload-management/workload-management-summary.md
@@ -1,55 +1,36 @@
----
-{
-"title": "Workload Management Overview",
-"language": "en"
-}
----
-
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-The proper allocation of a database's system resources (such as CPU, memory,
I/O, and storage) is crucial. By precisely managing these resources, it is
ensured that queries receive adequate resource support during execution,
thereby enhancing the overall performance and maintaining the stability of the
system. Doris provides resource control functionality, allowing users to divide
the physical resources within the cluster into multiple resource units. This
resource allocation control pre [...]
-
-## Scenarios for Resource Control
-Doris's resource control feature provides refined resource allocation and
management support for concurrent multi-user and multi-business scenarios,
ensuring efficient and stable system operation in complex environments.
Resource control is typically used in the following scenarios:
-- When different business data is stored in a single Doris cluster, resource
control prevents resource contention between different businesses.
-- When different users access the same data, different tenants are isolated to
avoid resource contention among users.
-- In scenarios with multiple concurrent workloads, resource queues adjust
query priorities to prevent faster queries from being blocked by longer-running
ones.
-- Implementing load circuit breakers that automatically terminate workloads
when circuit breaker conditions are met.
-
-## Methods for Resource Group Division
-In Doris, users can divide system resources into different resource pools
based on specific business needs, ensuring resource independence and mutual
non-interference between pools. By binding resource pools to different users,
Doris achieves refined resource management and control among users. Doris
offers three primary methods for resource division:
-- Resource Group: Uses BE nodes as the smallest granularity and divides
resources into multiple groups by setting tags.
-- Workload Group: Divides resources into multiple groups with CPU/Memory/IO as
the smallest granularity, enabling more detailed resource allocatio
-- Compute Group: A method for dividing resources in a storage-compute
separation model, similar to Resource Group, using BE nodes as the smallest
granularity.
-
-
-| Resource Group Division | Granularity of Isolation | Description
of Isolation Solution
| Soft/Hard Limit |
Cross-Group Queries|
-| ---------- | -----------
|---------------------------------------------------------------------------------------------------------|-----|-----|
-| Resource Group | Server node level, fully isolated resources | Sets
tags for BE nodes, with BE nodes sharing the same tag forming a resource group,
enabling node-level resource isolation within the cluster and resource
restriction per query. Facilitates fault isolation, where issues within one
resource group do not affect others. | Hard Limit |Not supported. Must
ensure at least one data replica within the resource group. |
-| Workload Group | Resource granularity, including CPU, memory, IO, etc.
| Utilizes cgroup for CPU, memory, IO, and concurrency resource control;
provides query queuing to avoid resource contention.
| Supports both hard and soft limits | Supported
|
-|Compute Group | Server node level, fully isolated resources |
Based on a storage-compute separation architecture, data is visible across
multiple Compute Groups; enables node-level resource isolation within the
cluster and resource restriction per query without relying on replica tags. |
Hard Limit | Not supported. Must ensure at least one data replica within
the resource group. |
-
-## Soft and Hard Resource Limits
-In Doris, soft limits (soft limit) and hard limits (hard limit) can be used to
restrict resource usage:
-- Hard Limit: Represents the absolute upper limit of resources that can be
used, which cannot be exceeded by tenants. Once the hard limit is reached,
requests for additional resources will be rejected. Hard limits are generally
used to prevent resource exhaustion within the cluster or contention between
different businesses, ensuring cluster stability and performance.
-- Soft Limit: Represents a recommended upper limit that can be exceeded. When
the system is not busy, tenants can borrow resources from other resource groups
if they exceed the soft limit. However, during peak times or when resources are
contended, tenants exceeding the soft limit will not receive additional
resources.
-
-Resource groups divided using Resource Group/Compute Group methods only
support hard limits. Workload Groups support both soft and hard limits. Soft
limits for Workload resources are typically used for managing sudden resource
spikes, such as temporary query peaks or brief data ingestion increases. Hard
limits are used to prevent resource abuse, enabling absolute resource isolation
between different businesses to avoid contention. Reasonable use of soft and
hard limits can prevent cluste [...]
+Workload management is an very important feature of Doris, playing a critical
role in the overall operation of the system. Through reasonable workload
management strategies, resource utilization can be optimized, system stability
enhanced, and response time reduced. It has the following abilities:
+
+- Resource Isolation: By dividing into multiple groups and setting resource
(CPU, Memory, IO) limits for each group, it ensures that there is no
interference between multiple users or different tasks (such as read and write
operations) of the same user.
+
+- Concurrency Control and Queuing: It can limit the number of tasks that can
be executed simultaneously in the entire cluster. When threshold is exceeded,
tasks will be queued.
+
+- Circuit Breaker: During query planning phase or execution phase, tasks can
be automatically cancelled based on conditions such as the estimated number of
partitions to be read, the amount of data to be scanned, the allocated memory
size, and the execution time. This avoids unreasonable tasks from occupying too
many system resources.
+
+
+## Resource Isolation Methods
+
+Doris can divide resource in the following three ways:
+
+- Resource Group: Divide multiple BE processes into groups setting tag for
each BE.
+
+- Workload Group: The resource (CPU, Memory, IO) within a BE are divided into
multiple resource groups through Cgroup, enabling more fine-grained resource
isolation.
+
+- Compute Group: It is a way of resource partitioning method in
compute-storage decoupled mode. Similar to Resource Group, it also takes BE as
the minimum granularity to divide multiple groups.
+
+The following table records the characteristics and advantageous scenarios of
different resource partitioning methods:
+
+| Resource Isolation Method | Isolation Granularity | Soft/Hard
Limit | Cross Resource Group Query |
+| ---------- | ----------- |-----|-----|
+| Resource Group | BE node level, with complete resource isolation, can
isolate BE failures | Hard limit |Not support. And it is necessary to
ensure that at least one copy of data is stored within the resource group. |
+| Workload Group | Isolation within BE process; cannot isolate BE failures
| Both hard and soft limit
| Support |
+|Compute Group | BE node level, with complete resource isolation,
can isolate BE failures | Hard limit | Not support |
+
+## Soft Limit and Hard Limit
+
+
+- Hard Limit: The hard limit refers to the absolute upper limit of resource
usage that tenants cannot exceed. Once the hard limit is reached, resource
requests for the excess part will be rejected. Hard limit are generally used to
prevent the depletion of resources within the cluster or resource preemption
between different businesses, ensuring the stability and performance of the
cluster.
+
+- Soft Limit: The soft limit is a resource limit that can be exceeded, usually
representing the recommended upper limit of resource usage. When the system is
not busy, if a tenant requests more resources than the soft limit, it can
borrow resources from other groups. When the system is busy and there is
resource contention, if a tenant requests resources exceeding the soft limit,
it will not be able to obtain additional resources.
+
+When using the Resource Group / Compute Group method to partition resources,
only the hard limit mode is supported. When using the Workload Group method to
partition resources, both the soft limit and hard limit of Workload Group are
supported. The soft limit of Workload Group is usually used for sudden resource
control, such as temporary query peaks or short-term increases in data writing.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
