This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new dcd31175e75 update log storage docs (#1252)
dcd31175e75 is described below
commit dcd31175e75b8bda1ad7893d246f9588174ed14e
Author: Kang <[email protected]>
AuthorDate: Fri Nov 1 23:53:55 2024 +0800
update log storage docs (#1252)
1. delete single replica load and compaction configs
2. add zstd compression config
3. fix format error for JSON, * and surround formula with quote
4. reference code in text description
5. change demo compression ratio from 7 to 5
# Versions
- [x] dev
- [x] 3.0
- [x] 2.1
- [x] 2.0
# Languages
- [x] Chinese
- [x] English
---
.../tutorials/log-storage-analysis.md | 115 ++++++++--------
gettingStarted/tutorials/log-storage-analysis.md | 138 +++++++++----------
.../practical-guide/log-storage-analysis.md | 148 ++++++++++-----------
.../practical-guide/log-storage-analysis.md | 138 +++++++++----------
.../practical-guide/log-storage-analysis.md | 138 +++++++++----------
5 files changed, 334 insertions(+), 343 deletions(-)
diff --git a/common_docs_zh/gettingStarted/tutorials/log-storage-analysis.md
b/common_docs_zh/gettingStarted/tutorials/log-storage-analysis.md
index 25797007642..d54c97a408e 100644
--- a/common_docs_zh/gettingStarted/tutorials/log-storage-analysis.md
+++ b/common_docs_zh/gettingStarted/tutorials/log-storage-analysis.md
@@ -77,19 +77,19 @@ Apache Doris 支持标准 SQL、兼容 MySQL 协议和语法，因此基于 Apac
```JSON
{
-"timestamp": "2014-03-08T00:50:03.8432810Z",
-"source": "ADOPTIONCUSTOMERS81",
-"node": "Engine000000000405",
-"level": "Information",
-"component": "DOWNLOADER",
-"clientRequestId": "671db15d-abad-94f6-dd93-b3a2e6000672",
-"message": "Downloading file path:
benchmark/2014/ADOPTIONCUSTOMERS81_94_0.parquet.gz",
-"properties": {
-"size": 1495636750,
-"format": "parquet",
-"rowCount": 855138,
-"downloadDuration": "00:01:58.3520561"
-}
+ "timestamp": "2014-03-08T00:50:03.8432810Z",
+ "source": "ADOPTIONCUSTOMERS81",
+ "node": "Engine000000000405",
+ "level": "Information",
+ "component": "DOWNLOADER",
+ "clientRequestId": "671db15d-abad-94f6-dd93-b3a2e6000672",
+ "message": "Downloading file path:
benchmark/2014/ADOPTIONCUSTOMERS81_94_0.parquet.gz",
+ "properties": {
+ "size": 1495636750,
+ "format": "parquet",
+ "rowCount": 855138,
+ "downloadDuration": "00:01:58.3520561"
+ }
}
```
@@ -112,43 +112,43 @@ Apache Doris 对 Flexible Schema 的日志数据提供了几个方面的支持
1. **评估写入资源**：计算公式如下：
- - 日增数据量 / 86400 s = 平均写入吞吐
- - 平均写入吞吐 x 写入吞吐峰值 / 均值比 = 峰值写入吞吐
- - 峰值写入吞吐 / 单核写入吞吐 = 峰值写入所需 CPU 核数
+ - `平均写入吞吐 = 日增数据量 / 86400 s`
+ - `峰值写入吞吐 = 平均写入吞吐 * 写入吞吐峰值 / 均值比`
+ - `峰值写入所需 CPU 核数 = 峰值写入吞吐 / 单核写入吞吐`
-2. **评估存储资源**：计算公式为「日增数据量 / 压缩率 * 副本数 * 数据存储周期 = 所需存储空间」。
+2. **评估存储资源**：计算公式为 `所需存储空间 = 日增数据量 / 压缩率 * 副本数 * 数据存储周期`
3. **评估查询资源**：查询的资源消耗随查询量和复杂度而异，建议初始预留 50% 的 CPU 资源用于查询，再根据实际测试情况进行调整。
4. **汇总整合资源**：由第 1 步和第 3 步估算出所需 CPU 核数后，除以单机 CPU 核数，估算出 BE 服务器数量，再根据 BE
服务器数量和第 2 步的结果，估算出每台 BE 服务器所需存储空间，然后分摊到 4～12 块数据盘，计算出单盘存储容量。
-以每天新增 100 TB 数据量（压缩前）、7 倍压缩率、1 副本、热数据存储 3 天、冷数据存储 30 天、写入吞吐峰值 / 均值比
200%、单核写入吞吐 10 MB/s、查询预留 50% CPU 资源为例，可估算出：
+以每天新增 100 TB 数据量（压缩前）、5 倍压缩率、1 副本、热数据存储 3 天、冷数据存储 30 天、写入吞吐峰值 / 均值比
200%、单核写入吞吐 10 MB/s、查询预留 50% CPU 资源为例，可估算出：
- FE：3 台服务器，每台配置 16 核 CPU、64 GB 内存、1 块 100 GB SSD 盘
-- BE：15 台服务器，每台配置 32 核 CPU、256 GB 内存、8 块 500 GB SSD 盘
-- Amazon S3 存储空间：即为预估冷数据存储空间，430 TB
+- BE：15 台服务器，每台配置 32 核 CPU、256 GB 内存、10 块 600 GB SSD 盘
+- S3 对象存储空间：即为预估冷数据存储空间，600 TB
该例子中，各关键指标的值及具体计算方法可见下表：
| 关键指标（单位） | 值 | 说明
|
| :------------------------------- | :---- |
:----------------------------------------------------------- |
| 日增数据量（TB） | 100 | 根据实际需求填写
|
-| 压缩率 | 7 | 一般为 5～10 倍（含索引），根据实际需求填写
|
+| 压缩率 | 5 | 一般为 3～10 倍（含索引），根据实际需求填写
|
| 副本数 | 1 | 根据实际需求填写，默认 1 副本，可选值：1，2，3
|
| 热数据存储周期（天） | 3 | 根据实际需求填写
|
| 冷数据存储周期（天） | 30 | 根据实际需求填写
|
-| 总存储周期（天） | 33 | 算法：热数据存储周期 + 冷数据存储周期
|
-| 预估热数据存储空间（TB） | 42.9 | 算法：日增数据量 / 压缩率 * 副本数 * 热数据存储周期 |
-| 预估冷数据存储空间（TB） | 428.6 | 算法：日增数据量 / 压缩率 * 副本数 * 冷数据存储周期 |
+| 总存储周期（天） | 33 | 算法：`热数据存储周期 + 冷数据存储周期`
|
+| 预估热数据存储空间（TB） | 60 | 算法：`日增数据量 / 压缩率 * 副本数 * 热数据存储周期` |
+| 预估冷数据存储空间（TB） | 600 | 算法：`日增数据量 / 压缩率 * 副本数 * 冷数据存储周期` |
| 写入吞吐峰值 / 均值比 | 200% | 根据实际需求填写，默认 200%
|
| 单机 CPU 核数 | 32 | 根据实际需求填写，默认 32 核
|
-| 平均写入吞吐（MB/s） | 1214 | 算法：日增数据量 / 86400 s
|
-| 峰值写入吞吐（MB/s） | 2427 | 算法：平均写入吞吐 * 写入吞吐峰值 / 均值比
|
-| 峰值写入所需 CPU 核数 | 242.7 | 算法：峰值写入吞吐 / 单机写入吞吐
|
+| 平均写入吞吐（MB/s） | 1214 | 算法：`日增数据量 / 86400 s`
|
+| 峰值写入吞吐（MB/s） | 2427 | 算法：`平均写入吞吐 * 写入吞吐峰值 / 均值比`
|
+| 峰值写入所需 CPU 核数 | 242.7 | 算法：`峰值写入吞吐 / 单核写入吞吐`
|
| 查询预留 CPU 百分比 | 50% | 根据实际需求填写，默认 50%
|
-| 预估 BE 服务器数 | 15.2 | 算法：峰值写入所需 CPU 核数 / 单机 CPU 核数 /（1 - 查询预留
CPU 百分比） |
-| 预估 BE 服务器数取整 | 15 | 算法：MAX (副本数，预估 BE 服务器数上取整)
|
-| 预估每台 BE 服务器存储空间（TB） | 4.03 | 算法：预估热数据存储空间 / 预估 BE 服务器数 /（1 - 30%），其中，30%
是存储空间预留值。建议每台 BE 服务器挂载 4～12 块数据盘，以提高 I/O 能力。 |
+| 预估 BE 服务器数 | 15.2 | 算法：`峰值写入所需 CPU 核数 / 单机 CPU 核数 /（1 -
查询预留 CPU 百分比）` |
+| 预估 BE 服务器数取整 | 15 | 算法：`MAX (副本数，预估 BE 服务器数取整)`
|
+| 预估每台 BE 服务器存储空间（TB） | 5.7 | 算法：`预估热数据存储空间 / 预估 BE 服务器数 /（1 - 30%）`，其中，30%
是存储空间预留值。建议每台 BE 服务器挂载 4～12 块数据盘，以提高 I/O 能力。 |
### 第 2 步：部署集群
@@ -168,7 +168,6 @@ Apache Doris 对 Flexible Schema 的日志数据提供了几个方面的支持
| `streaming_label_keep_max_second = 3600` `label_keep_max_second = 7200` |
高频导入事务标签内存占用多，保留时间调短。 |
| `enable_round_robin_create_tablet = true` | 创建 Tablet
时，采用 Round Robin 策略，尽量均匀。 |
| `tablet_rebalancer_type = partition` | 均衡 Tablet
时，采用每个分区内尽量均匀的策略。 |
-| `enable_single_replica_load = true` |
开启单副本导入，多个副本只需构建一次索引，减少 CPU 消耗。 |
| `autobucket_min_buckets = 10` | 将自动分桶的最小分桶数从
1 调大到 10，避免日志量增加时分桶不够。 |
| `max_backend_heartbeat_failure_tolerance_count = 10` | 日志场景下 BE
服务器压力较大，可能短时间心跳超时，因此将容忍次数从 1 调大到 10。 |
@@ -185,7 +184,6 @@ Apache Doris 对 Flexible Schema 的日志数据提供了几个方面的支持
| - | `file_cache_path = [{"path": "/mnt/datadisk0/file_cache",
"total_size":53687091200, "query_limit": "10737418240"},{"path":
"/mnt/datadisk1/file_cache", "total_size":53687091200,"query_limit":
"10737418240"}]` | 配置冷数据的缓存路径和相关设置，具体配置说明如下：<br />`path`：缓存路径<br
/>`total_size`：该缓存路径的总大小，单位为字节，53687091200 字节等于 50 GB<br
/>`query_limit`：单次查询可以从缓存路径中查询的最大数据量，单位为字节，10737418240 字节等于 10 GB |
| 写入 | `write_buffer_size = 1073741824` |
增加写入缓冲区（buffer）的文件大小，减少小文件和随机 I/O 操作，提升性能。 |
| - | `max_tablet_version_num = 20000` |
配合建表的 time_series compaction 策略，允许更多版本暂时未合并。 |
-| - | `enable_single_replica_load = true` |
开启单副本写入，减少 CPU 消耗。与 FE 配置相同。 |
| Compaction | `max_cumu_compaction_threads = 8` |
设置为 CPU 核数 / 4，意味着 CPU 资源的 1/4 用于写入，1/4 用于后台 Compaction，2/1 留给查询和其他操作。 |
| - | `inverted_index_compaction_enable = true` |
开启索引合并（index compaction），减少 Compaction 时的 CPU 消耗。 |
| - | `enable_segcompaction = false` `enable_ordered_data_compaction
= false` | 关闭日志场景不需要的两个 Compaction 功能。 |
@@ -209,35 +207,38 @@ Apache Doris 对 Flexible Schema 的日志数据提供了几个方面的支持
**配置分区分桶参数**
-- 分区时，按照以下说明配置：
-- 使用时间字段上的 [Range
分区](../../table-design/data-partition/#range-%E5%88%86%E5%8C%BA)，并开启
[动态分区](../../table-design/data-partition?_highlight=%E8%87%AA%E5%8A%A8&_highlight=%E5%88%86&_highlight=%E6%A1%B6#%E5%8A%A8%E6%80%81%E5%88%86%E5%8C%BA)，按天自动管理分区。
-- 使用 Datetime 类型的时间字段作为 Key，在查询最新 N 条日志时有数倍加速。
-- 分桶时，按照以下说明配置：
-- 分桶数量大致为集群磁盘总数的 3 倍。
-- 使用 Random 策略，配合写入时的 Single Tablet 导入，可以提升批量（Batch）写入的效率。
+分区时，按照以下说明配置：
+- 使用时间字段上的 [Range
分区](../../table-design/data-partition/#range-%E5%88%86%E5%8C%BA) (`PARTITION BY
RANGE(`ts`)`)，并开启 [动态分区](../../table-design/data-partition)
(`"dynamic_partition.enable" = "true"`)，按天自动管理分区。
+- 使用 Datetime 类型的时间字段作为 Key (`DUPLICATE KEY(ts)`)，在查询最新 N 条日志时有数倍加速。
+
+分桶时，按照以下说明配置：
+- 分桶数量大致为集群磁盘总数的 3 倍，每个桶的数据量压缩后 5GB 左右。
+- 使用 Random 策略 (`DISTRIBUTED BY RANDOM BUCKETS 60`)，配合写入时的 Single Tablet
导入，可以提升批量（Batch）写入的效率。
更多关于分区分桶的信息，可参考 [分区分桶](../../table-design/data-partition)。
+**配置压缩参数**
+- 使用 zstd 压缩算法(`"compression" = "zstd"`), 提高数据压缩率。
+
**配置 Compaction 参数**
按照以下说明配置 Compaction 参数：
-- 使用 time_series 策略，以减轻写放大效应，对于高吞吐日志写入的资源写入很重要。
-- 使用单副本 Compaction，减少多副本 Compaction 的开销。
+- 使用 time_series 策略(`"compaction_policy" =
"time_series"`)，以减轻写放大效应，对于高吞吐日志写入的资源写入很重要。
**建立和配置索引参数**
按照以下说明操作：
-- 对经常查询的字段建立索引。
+- 对经常查询的字段建立索引 (`USING INVERTED`)。
- 对需要全文检索的字段，将分词器（parser）参数赋值为 unicode，一般能满足大部分需求。如有支持短语查询的需求，将 support_phrase
参数赋值为 true；如不需要，则设置为 false，以降低存储空间。
**配置存储策略**
按照以下说明操作：
-- 对于热存储数据，如果使用云盘，可配置 1 副本；如果使用物理盘，则至少配置 2 副本。
-- 配置 `log_s3` 的存储位置，并设置 `log_policy_3day` 冷热数据分层策略，即在超过 3 天后将数据冷却至 `log_s3`
指定的存储位置。可参考以下代码：
+- 对于热存储数据，如果使用云盘，可配置 1 副本；如果使用物理盘，则至少配置 2 副本 (`"replication_num" = "2"`)。
+- 配置 `log_s3` 的存储位置 (`CREATE RESOURCE "log_s3"`)，并设置 `log_policy_3day`
冷热数据分层策略 (`CREATE STORAGE POLICY log_policy_3day`)，即在超过 3 天后将数据冷却至 `log_s3`
指定的存储位置。可参考以下代码：
```sql
CREATE DATABASE log_db;
@@ -274,20 +275,20 @@ CREATE TABLE log_table
ENGINE = OLAP
DUPLICATE KEY(`ts`)
PARTITION BY RANGE(`ts`) ()
-DISTRIBUTED BY RANDOM BUCKETS 250
+DISTRIBUTED BY RANDOM BUCKETS 60
PROPERTIES (
-"compaction_policy" = "time_series",
-"dynamic_partition.enable" = "true",
-"dynamic_partition.create_history_partition" = "true",
-"dynamic_partition.time_unit" = "DAY",
-"dynamic_partition.start" = "-30",
-"dynamic_partition.end" = "1",
-"dynamic_partition.prefix" = "p",
-"dynamic_partition.buckets" = "250",
-"dynamic_partition.replication_num" = "2", -- 存算分离不需要
-"replication_num" = "2" -- 存算分离不需要
-"enable_single_replica_compaction" = "true", -- 存算分离不需要
-"storage_policy" = "log_policy_3day" -- 存算分离不需要
+ "compression" = "zstd",
+ "compaction_policy" = "time_series",
+ "dynamic_partition.enable" = "true",
+ "dynamic_partition.create_history_partition" = "true",
+ "dynamic_partition.time_unit" = "DAY",
+ "dynamic_partition.start" = "-30",
+ "dynamic_partition.end" = "1",
+ "dynamic_partition.prefix" = "p",
+ "dynamic_partition.buckets" = "60",
+ "dynamic_partition.replication_num" = "2", -- 存算分离不需要
+ "replication_num" = "2", -- 存算分离不需要
+ "storage_policy" = "log_policy_3day" -- 存算分离不需要
);
```
@@ -477,7 +478,7 @@ FROM KAFKA (
SHOW ROUTINE LOAD;
```
-更多关于 Kafka 配置和使用的说明，可参考 [Routine
Load](../../data-operate/import/routine-load-manual)。
+更多关于 Kafka 配置和使用的说明，可参考 [Routine
Load](../../data-operate/import/import-way/routine-load-manual)。
**使用自定义程序采集日志**
diff --git a/gettingStarted/tutorials/log-storage-analysis.md
b/gettingStarted/tutorials/log-storage-analysis.md
index f000b40bf25..22b79b1c206 100644
--- a/gettingStarted/tutorials/log-storage-analysis.md
+++ b/gettingStarted/tutorials/log-storage-analysis.md
@@ -86,26 +86,26 @@ Here is a typical example of a semi-structured log in JSON
format. The top-level
```JSON
{
-"timestamp": "2014-03-08T00:50:03.8432810Z",
-"source": "ADOPTIONCUSTOMERS81",
-"node": "Engine000000000405",
-"level": "Information",
-"component": "DOWNLOADER",
-"clientRequestId": "671db15d-abad-94f6-dd93-b3a2e6000672",
-"message": "Downloading file path:
benchmark/2014/ADOPTIONCUSTOMERS81_94_0.parquet.gz",
-"properties": {
-"size": 1495636750,
-"format": "parquet",
-"rowCount": 855138,
-"downloadDuration": "00:01:58.3520561"
-}
+ "timestamp": "2014-03-08T00:50:03.8432810Z",
+ "source": "ADOPTIONCUSTOMERS81",
+ "node": "Engine000000000405",
+ "level": "Information",
+ "component": "DOWNLOADER",
+ "clientRequestId": "671db15d-abad-94f6-dd93-b3a2e6000672",
+ "message": "Downloading file path:
benchmark/2014/ADOPTIONCUSTOMERS81_94_0.parquet.gz",
+ "properties": {
+ "size": 1495636750,
+ "format": "parquet",
+ "rowCount": 855138,
+ "downloadDuration": "00:01:58.3520561"
+ }
}
```
Apache Doris provides several aspects of support for Flexible Schema log data:
-- For minor changes to top-level fields, Light Schema Change can be used to
add or remove columns and to add or remove indexes, enabling schema changes to
be completed in seconds. When planning a log platform, users only need to
consider which fields need to be indexed.
+- For changes to top-level fields, Light Schema Change can be used to add or
remove columns and to add or remove indexes, enabling schema changes to be
completed in seconds. When planning a log platform, users only need to consider
which fields need to be indexed.
- For extension fields similar to properties, the native semi-structured data
type `VARIANT` is provided, which can write any JSON data, automatically
recognize field names and types in JSON, and automatically split frequently
occurring fields for columnar storage for subsequent analysis. Additionally,
`VARIANT` can create inverted indexes to accelerate internal field queries and
retrievals.
@@ -125,17 +125,17 @@ Before deploying the cluster, you need to estimate the
hardware resources requir
1. Estimate the resources for data writing by the following calculation
formulas:
-- Daily data increment / 86400 s = Average write throughput
+- `Average write throughput = Daily data increment / 86400 s`
-- Average write throughput \* Ratio of the peak write throughput to the
average write throughput = Peak write throughput
+- `Peak write throughput = Average write throughput \* Ratio of the peak write
throughput to the average write throughput`
-- Peak write throughput / Write throughput of a single-core CPU = Number of
CPU cores for the peak write throughput
+- `Number of CPU cores for the peak write throughput = Peak write throughput /
Write throughput of a single-core CPU`
-2. Estimate the resources for data storage by the calculation formula: Daily
data increment / Data compression ratio \* Number of data copies \* Data
storage duration = Data storage space.
+1. Estimate the resources for data storage by the calculation formula:
`Storage space = Daily data increment / Data compression ratio * Number of data
copies * Data storage duration`.
-3. Estimate the resources for data querying. The resources for data querying
depend on the query volume and complexity. It is recommended to reserve 50% of
CPU resources for data query initially and then adjust according to the actual
test results.
+2. Estimate the resources for data querying. The resources for data querying
depend on the query volume and complexity. It is recommended to reserve 50% of
CPU resources for data query initially and then adjust according to the actual
test results.
-4. Integrate the calculation results as follows:
+3. Integrate the calculation results as follows:
1. Divide the number of CPU cores calculated in Step 1 and Step 3 by the
number of CPU cores of a BE server, and you can get the number of BE servers.
@@ -143,39 +143,39 @@ Before deploying the cluster, you need to estimate the
hardware resources requir
3. Allocate the storage space required for each BE server to 4 to 12 data
disks, and you can get the storage capacity required for a single data disk.
-For example, suppose that the daily data increment is 100 TB, the data
compression ratio is 7, the number of data copies is 1, the storage duration of
hot data is 3 days, the storage duration of cold data is 30 days, the ratio of
the peak write throughput to the average write throughput is 200%, the write
throughput of a single-core CUP is 10 MB/s, and 50% of CPU resources are
reserved for data querying, one can estimate that:
+For example, suppose that the daily data increment is 100 TB, the data
compression ratio is 5, the number of data copies is 1, the storage duration of
hot data is 3 days, the storage duration of cold data is 30 days, the ratio of
the peak write throughput to the average write throughput is 200%, the write
throughput of a single-core CUP is 10 MB/s, and 50% of CPU resources are
reserved for data querying, one can estimate that:
-- 3 FE servers are required, each configured with a 16-core CPU, 64 GB memory,
and an 1100 GB SSD disk.
+- 3 FE servers are required, each configured with a 16-core CPU, 64 GB memory,
and an 1 100 GB SSD disk.
-- 15 BE servers are required, each configured with a 32-core CPU, 256 GB
memory, and 8 500 GB SSD disks.
+- 15 BE servers are required, each configured with a 32-core CPU, 256 GB
memory, and 10 600 GB SSD disks.
-- S3:430 TB
+- S3 object storage space 600 TB
Refer to the following table to learn about the values of indicators in the
example above and how they are calculated.
| Indicator (Unit) | Value | Description |
| --- | --- | --- |
| Daily data increment (TB) | 100 | Specify the value according to your actual
needs. |
-| Data compression ratio | 7 | Specify the value according to your actual
needs, which is often between 5 to 10. Note that the data contains index data. |
+| Data compression ratio | 5 | Specify the value according to your actual
needs, which is often between 3 to 10. Note that the data contains index data. |
| Number of data copies | 1 | Specify the value according to your actual
needs, which can be 1, 2, or 3. The default value is 1. |
| Storage duration of hot data (day) | 3 | Specify the value according to
your actual needs. |
| Storage duration of cold data (day) | 30 | Specify the value according to
your actual needs. |
-| Data storage duration | 33 | Calculation formula: Storage duration of hot
data + Storage duration of cold data |
-| Estimated storage space for hot data (TB) | 42.9 | Calculation formula:
Daily data increment / Data compression ratios \* Number of data copies \*
Storage duration of hot data |
-| Estimated storage space for cold data (TB) | 428.6 | Calculation formula:
Daily data increment / Data compression ratios \* Number of data copies \*
Storage duration of cold data |
+| Data storage duration | 33 | Calculation formula: `Storage duration of hot
data + Storage duration of cold data` |
+| Estimated storage space for hot data (TB) | 60 | Calculation formula: `Daily
data increment / Data compression ratios * Number of data copies * Storage
duration of hot data` |
+| Estimated storage space for cold data (TB) | 600 | Calculation formula:
`Daily data increment / Data compression ratios * Number of data copies *
Storage duration of cold data` |
| Ratio of the peak write throughput to the average write throughput | 200% |
Specify the value according to your actual needs. The default value is 200%. |
| Number of CPU cores of a BE server | 32 | Specify the value according to
your actual needs. The default value is 32. |
-| Average write throughput (MB/s) | 1214 | Calculation formula: Daily data
increment / 86400 s |
-| Peak write throughput (MB/s) | 2427 | Calculation formula: Average write
throughput \* Ratio of the peak write throughput to the average write
throughput |
-| Number of CPU cores for the peak write throughput | 242.7 | Calculation
formula: Peak write throughput / Write throughput of a single-core CPU |
+| Average write throughput (MB/s) | 1214 | Calculation formula: `Daily data
increment / 86400 s` |
+| Peak write throughput (MB/s) | 2427 | Calculation formula: `Average write
throughput * Ratio of the peak write throughput to the average write
throughput` |
+| Number of CPU cores for the peak write throughput | 242.7 | Calculation
formula: `Peak write throughput / Write throughput of a single-core CPU` |
| Percent of CPU resources reserved for data querying | 50% | Specify the
value according to your actual needs. The default value is 50%. |
-| Estimated number of BE servers | 15.2 | Calculation formula: Number of CPU
cores for the peak write throughput / Number of CPU cores of a BE server /（1 -
Percent of CPU resources reserved for data querying） |
-| Rounded number of BE servers | 15 | Calculation formula: MAX (Number of
data copies, Estimated number of BE servers) |
-| Estimated data storage space for each BE server (TB) | 4.03 | Calculation
formula: Estimated storage space for hot data / Estimated number of BE servers
/（1 - 30%）, where 30% represents the percent of reserved storage space.<br
/><br />It is recommended to mount 4 to 12 data disks on each BE server to
enhance I/O capabilities. |
+| Estimated number of BE servers | 15.2 | Calculation formula: `Number of CPU
cores for the peak write throughput / Number of CPU cores of a BE server /(1 -
Percent of CPU resources reserved for data querying)` |
+| Rounded number of BE servers | 15 | Calculation formula: `MAX (Number of
data copies, Estimated number of BE servers)` |
+| Estimated data storage space for each BE server (TB) | 5.7 | Calculation
formula: `Estimated storage space for hot data / Estimated number of BE servers
/(1 - 30%)`, where 30% represents the percent of reserved storage space.<br
/><br />It is recommended to mount 4 to 12 data disks on each BE server to
enhance I/O capabilities. |
### Step 2: Deploy the cluster
-After estimating the resources, you need to deploy the cluster. It is
recommended to deploy in both physical and virtual environments manually. For
manual deployment, refer to [Manual
Deployment](../install/cluster-deployment/standard-deployment.md).
+After estimating the resources, you need to deploy the cluster. It is
recommended to deploy in both physical and virtual environments manually. For
manual deployment, refer to [Manual
Deployment](../../install/cluster-deployment/standard-deployment.md).
### Step 3: Optimize FE and BE configurations
@@ -191,11 +191,10 @@ You can find FE configuration fields in
`fe/conf/fe.conf`. Refer to the followin
| `streaming_label_keep_max_second = 3600` `label_keep_max_second = 7200` |
Increase the retention time to handle high-frequency import transactions with
high memory usage. |
| `enable_round_robin_create_tablet = true` | When creating
Tablets, use a Round Robin strategy to distribute evenly. |
| `tablet_rebalancer_type = partition` | When
balancing Tablets, use a strategy to evenly distribute within each partition. |
-| `enable_single_replica_load = true` | Enable
single-replica import, where multiple replicas only need to build an index once
to reduce CPU consumption. |
| `autobucket_min_buckets = 10` | Increase the
minimum number of automatically bucketed buckets from 1 to 10 to avoid
insufficient buckets when the log volume increases. |
| `max_backend_heartbeat_failure_tolerance_count = 10` | In log
scenarios, the BE server may experience high pressure, leading to short-term
timeouts, so increase the tolerance count from 1 to 10. |
-For more information, refer to [FE
Configuration](../admin-manual/config/fe-config.md).
+For more information, refer to [FE
Configuration](../../admin-manual/config/fe-config.md).
**Optimize BE configurations**
@@ -208,7 +207,6 @@ You can find BE configuration fields in `be/conf/be.conf`.
Refer to the followin
| - | `file_cache_path = [{"path": "/mnt/datadisk0/file_cache",
"total_size":53687091200, "query_limit": "10737418240"},{"path":
"/mnt/datadisk1/file_cache", "total_size":53687091200,"query_limit":
"10737418240"}]` | Configure the cache path and related settings for cold data
with the following specific configurations:<br/>`path`: cache
path<br/>`total_size`: total size of the cache path in bytes, where 53687091200
bytes equals 50 GB<br/>`query_limit`: maximum amount of data tha [...]
| Write | `write_buffer_size = 1073741824` |
Increase the file size of the write buffer to reduce small files and random I/O
operations, improving performance. |
| - | `max_tablet_version_num = 20000` |
In coordination with the time_series compaction strategy for table creation,
allow more versions to remain temporarily unmerged |
-| - | `enable_single_replica_load = true` |
Enable single replica import to reduce CPU consumption, consistent with FE
configuration. |
| Compaction | `max_cumu_compaction_threads = 8` |
Set to CPU core count / 4, indicating that 1/4 of CPU resources are used for
writing, 1/4 for background compaction, and 2/1 for queries and other
operations. |
| - | `inverted_index_compaction_enable = true` |
Enable inverted index compaction to reduce CPU consumption during compaction. |
| - | `enable_segcompaction = false` `enable_ordered_data_compaction
= false` | Disable two compaction features that are unnecessary for log
scenarios. |
@@ -225,7 +223,7 @@ You can find BE configuration fields in `be/conf/be.conf`.
Refer to the followin
| - | `trash_file_expire_time_sec = 300`
`path_gc_check_interval_second = 900` `path_scan_interval_second = 900` |
Accelerate the recycling of trash files. |
-For more information, refer to [BE
Configuration](../admin-manual/config/be-config.md).
+For more information, refer to [BE
Configuration](../../admin-manual/config/be-config.md).
### Step 4: Create tables
@@ -235,41 +233,43 @@ Due to the distinct characteristics of both writing and
querying log data, it is
- For data partitioning:
- - Enable [range
partitioning](https://doris.apache.org/docs/table-design/data-partition#range-partition)
with [dynamic
partitions](https://doris.apache.org/docs/table-design/data-partition#dynamic-partition)
managed automatically by day.
+ - Enable [range
partitioning](../../table-design/data-partition#range-partition) (`PARTITION BY
RANGE(`ts`)`) with [dynamic
partitions](../../table-design/data-partition#dynamic-partition)
(`"dynamic_partition.enable" = "true"`) managed automatically by day.
- - Use a field in the DATETIME type as the key for accelerated retrieval of
the latest N log entries.
+ - Use a field in the DATETIME type as the key (`DUPLICATE KEY(ts)`) for
accelerated retrieval of the latest N log entries.
- For data bucketing:
- - Configure the number of buckets to be roughly three times the total
number of disks in the cluster.
+ - Configure the number of buckets to be roughly three times the total
number of disks in the cluster, with each bucket containing approximately 5GB
of data after compression.
- - Use the Random strategy to optimize batch writing efficiency when paired
with single tablet imports.
+ - Use the Random strategy (`DISTRIBUTED BY RANDOM BUCKETS 60`) to optimize
batch writing efficiency when paired with single tablet imports.
-For more information, refer to [Data
Partitioning](../table-design/data-partition.md).
+For more information, refer to [Data
Partitioning](../../table-design/data-partition.md).
-**Configure compaction fileds**
+**Configure compression parameters**
-Configure compaction fields as follows:
+Use the zstd compression algorithm ("compression" = "zstd") to improve data
compression efficiency.
-- Use the time_series strategy to reduce write amplification, which is crucial
for high-throughput log writes.
+**Configure compaction parameters**
+
+Configure compaction fields as follows:
-- Use single-replica compaction to minimize the overhead of multi-replica
compaction.
+- Use the time_series strategy (`"compaction_policy" = "time_series"`) to
reduce write amplification, which is crucial for high-throughput log writes.
-**Configure index fields**
+**Configure index parameters**
Configuring index fields as follows:
-- Create indexes for fields that are frequently queried.
+- Create indexes for fields that are frequently queried (`USING INVERTED`).
- For fields that require full-text search, specify the parser field as
unicode, which satisfies most requirements. If there is a need to support
phrase queries, set the support_phrase field to true; if not needed, set it to
false to reduce storage space.
-**Configure storage policies**
+**Configure storage parameters**
Configure storage policies as follows:
-- For storage of hot data, if using cloud storage, configure the number of
data copies as 1; if using physical disks, configure the number of data copies
as at least 2.
+- For storage of hot data, if using cloud storage, configure the number of
data copies as 1; if using physical disks, configure the number of data copies
as at least 2 (`"replication_num" = "2"`).
-- Configure the storage location for log_s3 and set the log_policy_3day
policy, where the data is cooled and moved to the specified storage location of
log_s3 after 3 days. Refer to the code below.
+- Configure the storage location for log_s3 (`CREATE RESOURCE "log_s3"`) and
set the log_policy_3day policy (`CREATE STORAGE POLICY log_policy_3day`), where
the data is cooled and moved to the specified storage location of log_s3 after
3 days. Refer to the code below.
```SQL
CREATE DATABASE log_db;
@@ -306,20 +306,20 @@ CREATE TABLE log_table
ENGINE = OLAP
DUPLICATE KEY(`ts`)
PARTITION BY RANGE(`ts`) ()
-DISTRIBUTED BY RANDOM BUCKETS 250
+DISTRIBUTED BY RANDOM BUCKETS 60
PROPERTIES (
-"compaction_policy" = "time_series",
-"dynamic_partition.enable" = "true",
-"dynamic_partition.create_history_partition" = "true",
-"dynamic_partition.time_unit" = "DAY",
-"dynamic_partition.start" = "-30",
-"dynamic_partition.end" = "1",
-"dynamic_partition.prefix" = "p",
-"dynamic_partition.buckets" = "250",
-"dynamic_partition.replication_num" = "2", -- unneccessary for the
compute-storage coupled mode
-"replication_num" = "2" -- unneccessary for the compute-storage coupled mode
-"enable_single_replica_compaction" = "true", -- unneccessary for the
compute-storage coupled mode
-"storage_policy" = "log_policy_3day" -- unneccessary for the compute-storage
coupled mode
+ "compression" = "zstd",
+ "compaction_policy" = "time_series",
+ "dynamic_partition.enable" = "true",
+ "dynamic_partition.create_history_partition" = "true",
+ "dynamic_partition.time_unit" = "DAY",
+ "dynamic_partition.start" = "-30",
+ "dynamic_partition.end" = "1",
+ "dynamic_partition.prefix" = "p",
+ "dynamic_partition.buckets" = "60",
+ "dynamic_partition.replication_num" = "2", -- unneccessary for the
compute-storage coupled mode
+ "replication_num" = "2", -- unneccessary for the compute-storage coupled mode
+ "storage_policy" = "log_policy_3day" -- unneccessary for the compute-storage
coupled mode
);
```
@@ -396,7 +396,7 @@ output {
./bin/logstash -f logstash_demo.conf
```
-For more information about the Logstash Doris Output plugin, see [Logstash
Doris Output Plugin](../ecosystem/logstash.md).
+For more information about the Logstash Doris Output plugin, see [Logstash
Doris Output Plugin](../../ecosystem/logstash.md).
**Integrating Filebeat**
@@ -464,7 +464,7 @@ headers:
./filebeat-doris-1.0.0 -c filebeat_demo.yml
```
-For more information about Filebeat, refer to [Beats Doris Output
Plugin](../ecosystem/beats.md).
+For more information about Filebeat, refer to [Beats Doris Output
Plugin](../../ecosystem/beats.md).
**Integrating Kafka**
@@ -497,7 +497,7 @@ FROM KAFKA (
<br />SHOW ROUTINE LOAD;
```
-For more information about Kafka, see [Routine
Load](../data-operate/import/routine-load-manual.md)。
+For more information about Kafka, see [Routine
Load](../../data-operate/import/import-way/routine-load-manual.md).
**Using customized programs to collect logs**
diff --git a/versioned_docs/version-2.0/practical-guide/log-storage-analysis.md
b/versioned_docs/version-2.0/practical-guide/log-storage-analysis.md
index f000b40bf25..951cbc70423 100644
--- a/versioned_docs/version-2.0/practical-guide/log-storage-analysis.md
+++ b/versioned_docs/version-2.0/practical-guide/log-storage-analysis.md
@@ -86,36 +86,26 @@ Here is a typical example of a semi-structured log in JSON
format. The top-level
```JSON
{
-"timestamp": "2014-03-08T00:50:03.8432810Z",
-"source": "ADOPTIONCUSTOMERS81",
-"node": "Engine000000000405",
-"level": "Information",
-"component": "DOWNLOADER",
-"clientRequestId": "671db15d-abad-94f6-dd93-b3a2e6000672",
-"message": "Downloading file path:
benchmark/2014/ADOPTIONCUSTOMERS81_94_0.parquet.gz",
-"properties": {
-"size": 1495636750,
-"format": "parquet",
-"rowCount": 855138,
-"downloadDuration": "00:01:58.3520561"
-}
+ "timestamp": "2014-03-08T00:50:03.8432810Z",
+ "source": "ADOPTIONCUSTOMERS81",
+ "node": "Engine000000000405",
+ "level": "Information",
+ "component": "DOWNLOADER",
+ "clientRequestId": "671db15d-abad-94f6-dd93-b3a2e6000672",
+ "message": "Downloading file path:
benchmark/2014/ADOPTIONCUSTOMERS81_94_0.parquet.gz",
+ "properties": {
+ "size": 1495636750,
+ "format": "parquet",
+ "rowCount": 855138,
+ "downloadDuration": "00:01:58.3520561"
+ }
}
```
Apache Doris provides several aspects of support for Flexible Schema log data:
-- For minor changes to top-level fields, Light Schema Change can be used to
add or remove columns and to add or remove indexes, enabling schema changes to
be completed in seconds. When planning a log platform, users only need to
consider which fields need to be indexed.
-
-- For extension fields similar to properties, the native semi-structured data
type `VARIANT` is provided, which can write any JSON data, automatically
recognize field names and types in JSON, and automatically split frequently
occurring fields for columnar storage for subsequent analysis. Additionally,
`VARIANT` can create inverted indexes to accelerate internal field queries and
retrievals.
-
-Compared to Elasticsearch's Dynamic Mapping, Apache Doris's Flexible Schema
has the following advantages:
-
-- Allows a field to have multiple types, `VARIANT` automatically handles
conflicts and type promotion for fields, better adapting to iterative changes
in log data.
-
-- `VARIANT` automatically merges infrequently occurring fields into a column
store to avoid performance issues caused by excessive fields, metadata, or
columns.
-
-- Not only can columns be dynamically added, but they can also be dynamically
deleted, and indexes can be dynamically added or removed, eliminating the need
to index all fields at the beginning like Elasticsearch, reducing unnecessary
costs.
+- For changes to top-level fields, Light Schema Change can be used to add or
remove columns and to add or remove indexes, enabling schema changes to be
completed in seconds. When planning a log platform, users only need to consider
which fields need to be indexed.
## Operational guide
@@ -125,17 +115,17 @@ Before deploying the cluster, you need to estimate the
hardware resources requir
1. Estimate the resources for data writing by the following calculation
formulas:
-- Daily data increment / 86400 s = Average write throughput
+- `Average write throughput = Daily data increment / 86400 s`
-- Average write throughput \* Ratio of the peak write throughput to the
average write throughput = Peak write throughput
+- `Peak write throughput = Average write throughput \* Ratio of the peak write
throughput to the average write throughput`
-- Peak write throughput / Write throughput of a single-core CPU = Number of
CPU cores for the peak write throughput
+- `Number of CPU cores for the peak write throughput = Peak write throughput /
Write throughput of a single-core CPU`
-2. Estimate the resources for data storage by the calculation formula: Daily
data increment / Data compression ratio \* Number of data copies \* Data
storage duration = Data storage space.
+1. Estimate the resources for data storage by the calculation formula:
`Storage space = Daily data increment / Data compression ratio * Number of data
copies * Data storage duration`.
-3. Estimate the resources for data querying. The resources for data querying
depend on the query volume and complexity. It is recommended to reserve 50% of
CPU resources for data query initially and then adjust according to the actual
test results.
+2. Estimate the resources for data querying. The resources for data querying
depend on the query volume and complexity. It is recommended to reserve 50% of
CPU resources for data query initially and then adjust according to the actual
test results.
-4. Integrate the calculation results as follows:
+3. Integrate the calculation results as follows:
1. Divide the number of CPU cores calculated in Step 1 and Step 3 by the
number of CPU cores of a BE server, and you can get the number of BE servers.
@@ -143,39 +133,39 @@ Before deploying the cluster, you need to estimate the
hardware resources requir
3. Allocate the storage space required for each BE server to 4 to 12 data
disks, and you can get the storage capacity required for a single data disk.
-For example, suppose that the daily data increment is 100 TB, the data
compression ratio is 7, the number of data copies is 1, the storage duration of
hot data is 3 days, the storage duration of cold data is 30 days, the ratio of
the peak write throughput to the average write throughput is 200%, the write
throughput of a single-core CUP is 10 MB/s, and 50% of CPU resources are
reserved for data querying, one can estimate that:
+For example, suppose that the daily data increment is 100 TB, the data
compression ratio is 5, the number of data copies is 1, the storage duration of
hot data is 3 days, the storage duration of cold data is 30 days, the ratio of
the peak write throughput to the average write throughput is 200%, the write
throughput of a single-core CUP is 10 MB/s, and 50% of CPU resources are
reserved for data querying, one can estimate that:
-- 3 FE servers are required, each configured with a 16-core CPU, 64 GB memory,
and an 1100 GB SSD disk.
+- 3 FE servers are required, each configured with a 16-core CPU, 64 GB memory,
and an 1 100 GB SSD disk.
-- 15 BE servers are required, each configured with a 32-core CPU, 256 GB
memory, and 8 500 GB SSD disks.
+- 15 BE servers are required, each configured with a 32-core CPU, 256 GB
memory, and 10 600 GB SSD disks.
-- S3:430 TB
+- S3 object storage space 600 TB
Refer to the following table to learn about the values of indicators in the
example above and how they are calculated.
| Indicator (Unit) | Value | Description |
| --- | --- | --- |
| Daily data increment (TB) | 100 | Specify the value according to your actual
needs. |
-| Data compression ratio | 7 | Specify the value according to your actual
needs, which is often between 5 to 10. Note that the data contains index data. |
+| Data compression ratio | 5 | Specify the value according to your actual
needs, which is often between 3 to 10. Note that the data contains index data. |
| Number of data copies | 1 | Specify the value according to your actual
needs, which can be 1, 2, or 3. The default value is 1. |
| Storage duration of hot data (day) | 3 | Specify the value according to
your actual needs. |
| Storage duration of cold data (day) | 30 | Specify the value according to
your actual needs. |
-| Data storage duration | 33 | Calculation formula: Storage duration of hot
data + Storage duration of cold data |
-| Estimated storage space for hot data (TB) | 42.9 | Calculation formula:
Daily data increment / Data compression ratios \* Number of data copies \*
Storage duration of hot data |
-| Estimated storage space for cold data (TB) | 428.6 | Calculation formula:
Daily data increment / Data compression ratios \* Number of data copies \*
Storage duration of cold data |
+| Data storage duration | 33 | Calculation formula: `Storage duration of hot
data + Storage duration of cold data` |
+| Estimated storage space for hot data (TB) | 60 | Calculation formula: `Daily
data increment / Data compression ratios * Number of data copies * Storage
duration of hot data` |
+| Estimated storage space for cold data (TB) | 600 | Calculation formula:
`Daily data increment / Data compression ratios * Number of data copies *
Storage duration of cold data` |
| Ratio of the peak write throughput to the average write throughput | 200% |
Specify the value according to your actual needs. The default value is 200%. |
| Number of CPU cores of a BE server | 32 | Specify the value according to
your actual needs. The default value is 32. |
-| Average write throughput (MB/s) | 1214 | Calculation formula: Daily data
increment / 86400 s |
-| Peak write throughput (MB/s) | 2427 | Calculation formula: Average write
throughput \* Ratio of the peak write throughput to the average write
throughput |
-| Number of CPU cores for the peak write throughput | 242.7 | Calculation
formula: Peak write throughput / Write throughput of a single-core CPU |
+| Average write throughput (MB/s) | 1214 | Calculation formula: `Daily data
increment / 86400 s` |
+| Peak write throughput (MB/s) | 2427 | Calculation formula: `Average write
throughput * Ratio of the peak write throughput to the average write
throughput` |
+| Number of CPU cores for the peak write throughput | 242.7 | Calculation
formula: `Peak write throughput / Write throughput of a single-core CPU` |
| Percent of CPU resources reserved for data querying | 50% | Specify the
value according to your actual needs. The default value is 50%. |
-| Estimated number of BE servers | 15.2 | Calculation formula: Number of CPU
cores for the peak write throughput / Number of CPU cores of a BE server /（1 -
Percent of CPU resources reserved for data querying） |
-| Rounded number of BE servers | 15 | Calculation formula: MAX (Number of
data copies, Estimated number of BE servers) |
-| Estimated data storage space for each BE server (TB) | 4.03 | Calculation
formula: Estimated storage space for hot data / Estimated number of BE servers
/（1 - 30%）, where 30% represents the percent of reserved storage space.<br
/><br />It is recommended to mount 4 to 12 data disks on each BE server to
enhance I/O capabilities. |
+| Estimated number of BE servers | 15.2 | Calculation formula: `Number of CPU
cores for the peak write throughput / Number of CPU cores of a BE server /(1 -
Percent of CPU resources reserved for data querying)` |
+| Rounded number of BE servers | 15 | Calculation formula: `MAX (Number of
data copies, Estimated number of BE servers)` |
+| Estimated data storage space for each BE server (TB) | 5.7 | Calculation
formula: `Estimated storage space for hot data / Estimated number of BE servers
/(1 - 30%)`, where 30% represents the percent of reserved storage space.<br
/><br />It is recommended to mount 4 to 12 data disks on each BE server to
enhance I/O capabilities. |
### Step 2: Deploy the cluster
-After estimating the resources, you need to deploy the cluster. It is
recommended to deploy in both physical and virtual environments manually. For
manual deployment, refer to [Manual
Deployment](../install/cluster-deployment/standard-deployment.md).
+After estimating the resources, you need to deploy the cluster. It is
recommended to deploy in both physical and virtual environments manually. For
manual deployment, refer to [Manual
Deployment](../../install/cluster-deployment/standard-deployment.md).
### Step 3: Optimize FE and BE configurations
@@ -191,11 +181,10 @@ You can find FE configuration fields in
`fe/conf/fe.conf`. Refer to the followin
| `streaming_label_keep_max_second = 3600` `label_keep_max_second = 7200` |
Increase the retention time to handle high-frequency import transactions with
high memory usage. |
| `enable_round_robin_create_tablet = true` | When creating
Tablets, use a Round Robin strategy to distribute evenly. |
| `tablet_rebalancer_type = partition` | When
balancing Tablets, use a strategy to evenly distribute within each partition. |
-| `enable_single_replica_load = true` | Enable
single-replica import, where multiple replicas only need to build an index once
to reduce CPU consumption. |
| `autobucket_min_buckets = 10` | Increase the
minimum number of automatically bucketed buckets from 1 to 10 to avoid
insufficient buckets when the log volume increases. |
| `max_backend_heartbeat_failure_tolerance_count = 10` | In log
scenarios, the BE server may experience high pressure, leading to short-term
timeouts, so increase the tolerance count from 1 to 10. |
-For more information, refer to [FE
Configuration](../admin-manual/config/fe-config.md).
+For more information, refer to [FE
Configuration](../../admin-manual/config/fe-config.md).
**Optimize BE configurations**
@@ -208,7 +197,6 @@ You can find BE configuration fields in `be/conf/be.conf`.
Refer to the followin
| - | `file_cache_path = [{"path": "/mnt/datadisk0/file_cache",
"total_size":53687091200, "query_limit": "10737418240"},{"path":
"/mnt/datadisk1/file_cache", "total_size":53687091200,"query_limit":
"10737418240"}]` | Configure the cache path and related settings for cold data
with the following specific configurations:<br/>`path`: cache
path<br/>`total_size`: total size of the cache path in bytes, where 53687091200
bytes equals 50 GB<br/>`query_limit`: maximum amount of data tha [...]
| Write | `write_buffer_size = 1073741824` |
Increase the file size of the write buffer to reduce small files and random I/O
operations, improving performance. |
| - | `max_tablet_version_num = 20000` |
In coordination with the time_series compaction strategy for table creation,
allow more versions to remain temporarily unmerged |
-| - | `enable_single_replica_load = true` |
Enable single replica import to reduce CPU consumption, consistent with FE
configuration. |
| Compaction | `max_cumu_compaction_threads = 8` |
Set to CPU core count / 4, indicating that 1/4 of CPU resources are used for
writing, 1/4 for background compaction, and 2/1 for queries and other
operations. |
| - | `inverted_index_compaction_enable = true` |
Enable inverted index compaction to reduce CPU consumption during compaction. |
| - | `enable_segcompaction = false` `enable_ordered_data_compaction
= false` | Disable two compaction features that are unnecessary for log
scenarios. |
@@ -225,7 +213,7 @@ You can find BE configuration fields in `be/conf/be.conf`.
Refer to the followin
| - | `trash_file_expire_time_sec = 300`
`path_gc_check_interval_second = 900` `path_scan_interval_second = 900` |
Accelerate the recycling of trash files. |
-For more information, refer to [BE
Configuration](../admin-manual/config/be-config.md).
+For more information, refer to [BE
Configuration](../../admin-manual/config/be-config.md).
### Step 4: Create tables
@@ -235,41 +223,43 @@ Due to the distinct characteristics of both writing and
querying log data, it is
- For data partitioning:
- - Enable [range
partitioning](https://doris.apache.org/docs/table-design/data-partition#range-partition)
with [dynamic
partitions](https://doris.apache.org/docs/table-design/data-partition#dynamic-partition)
managed automatically by day.
+ - Enable [range
partitioning](../../table-design/data-partition#range-partition) (`PARTITION BY
RANGE(`ts`)`) with [dynamic
partitions](../../table-design/data-partition#dynamic-partition)
(`"dynamic_partition.enable" = "true"`) managed automatically by day.
- - Use a field in the DATETIME type as the key for accelerated retrieval of
the latest N log entries.
+ - Use a field in the DATETIME type as the key (`DUPLICATE KEY(ts)`) for
accelerated retrieval of the latest N log entries.
- For data bucketing:
- - Configure the number of buckets to be roughly three times the total
number of disks in the cluster.
+ - Configure the number of buckets to be roughly three times the total
number of disks in the cluster, with each bucket containing approximately 5GB
of data after compression.
- - Use the Random strategy to optimize batch writing efficiency when paired
with single tablet imports.
+ - Use the Random strategy (`DISTRIBUTED BY RANDOM BUCKETS 60`) to optimize
batch writing efficiency when paired with single tablet imports.
-For more information, refer to [Data
Partitioning](../table-design/data-partition.md).
+For more information, refer to [Data
Partitioning](../../table-design/data-partition.md).
-**Configure compaction fileds**
+**Configure compression parameters**
-Configure compaction fields as follows:
+Use the zstd compression algorithm ("compression" = "zstd") to improve data
compression efficiency.
-- Use the time_series strategy to reduce write amplification, which is crucial
for high-throughput log writes.
+**Configure compaction parameters**
+
+Configure compaction fields as follows:
-- Use single-replica compaction to minimize the overhead of multi-replica
compaction.
+- Use the time_series strategy (`"compaction_policy" = "time_series"`) to
reduce write amplification, which is crucial for high-throughput log writes.
-**Configure index fields**
+**Configure index parameters**
Configuring index fields as follows:
-- Create indexes for fields that are frequently queried.
+- Create indexes for fields that are frequently queried (`USING INVERTED`).
- For fields that require full-text search, specify the parser field as
unicode, which satisfies most requirements. If there is a need to support
phrase queries, set the support_phrase field to true; if not needed, set it to
false to reduce storage space.
-**Configure storage policies**
+**Configure storage parameters**
Configure storage policies as follows:
-- For storage of hot data, if using cloud storage, configure the number of
data copies as 1; if using physical disks, configure the number of data copies
as at least 2.
+- For storage of hot data, if using cloud storage, configure the number of
data copies as 1; if using physical disks, configure the number of data copies
as at least 2 (`"replication_num" = "2"`).
-- Configure the storage location for log_s3 and set the log_policy_3day
policy, where the data is cooled and moved to the specified storage location of
log_s3 after 3 days. Refer to the code below.
+- Configure the storage location for log_s3 (`CREATE RESOURCE "log_s3"`) and
set the log_policy_3day policy (`CREATE STORAGE POLICY log_policy_3day`), where
the data is cooled and moved to the specified storage location of log_s3 after
3 days. Refer to the code below.
```SQL
CREATE DATABASE log_db;
@@ -306,20 +296,20 @@ CREATE TABLE log_table
ENGINE = OLAP
DUPLICATE KEY(`ts`)
PARTITION BY RANGE(`ts`) ()
-DISTRIBUTED BY RANDOM BUCKETS 250
+DISTRIBUTED BY RANDOM BUCKETS 60
PROPERTIES (
-"compaction_policy" = "time_series",
-"dynamic_partition.enable" = "true",
-"dynamic_partition.create_history_partition" = "true",
-"dynamic_partition.time_unit" = "DAY",
-"dynamic_partition.start" = "-30",
-"dynamic_partition.end" = "1",
-"dynamic_partition.prefix" = "p",
-"dynamic_partition.buckets" = "250",
-"dynamic_partition.replication_num" = "2", -- unneccessary for the
compute-storage coupled mode
-"replication_num" = "2" -- unneccessary for the compute-storage coupled mode
-"enable_single_replica_compaction" = "true", -- unneccessary for the
compute-storage coupled mode
-"storage_policy" = "log_policy_3day" -- unneccessary for the compute-storage
coupled mode
+ "compression" = "zstd",
+ "compaction_policy" = "time_series",
+ "dynamic_partition.enable" = "true",
+ "dynamic_partition.create_history_partition" = "true",
+ "dynamic_partition.time_unit" = "DAY",
+ "dynamic_partition.start" = "-30",
+ "dynamic_partition.end" = "1",
+ "dynamic_partition.prefix" = "p",
+ "dynamic_partition.buckets" = "60",
+ "dynamic_partition.replication_num" = "2", -- unneccessary for the
compute-storage coupled mode
+ "replication_num" = "2", -- unneccessary for the compute-storage coupled mode
+ "storage_policy" = "log_policy_3day" -- unneccessary for the compute-storage
coupled mode
);
```
@@ -396,7 +386,7 @@ output {
./bin/logstash -f logstash_demo.conf
```
-For more information about the Logstash Doris Output plugin, see [Logstash
Doris Output Plugin](../ecosystem/logstash.md).
+For more information about the Logstash Doris Output plugin, see [Logstash
Doris Output Plugin](../../ecosystem/logstash.md).
**Integrating Filebeat**
@@ -464,7 +454,7 @@ headers:
./filebeat-doris-1.0.0 -c filebeat_demo.yml
```
-For more information about Filebeat, refer to [Beats Doris Output
Plugin](../ecosystem/beats.md).
+For more information about Filebeat, refer to [Beats Doris Output
Plugin](../../ecosystem/beats.md).
**Integrating Kafka**
@@ -497,7 +487,7 @@ FROM KAFKA (
<br />SHOW ROUTINE LOAD;
```
-For more information about Kafka, see [Routine
Load](../data-operate/import/routine-load-manual.md)。
+For more information about Kafka, see [Routine
Load](../../data-operate/import/import-way/routine-load-manual.md).
**Using customized programs to collect logs**
diff --git a/versioned_docs/version-2.1/practical-guide/log-storage-analysis.md
b/versioned_docs/version-2.1/practical-guide/log-storage-analysis.md
index f000b40bf25..22b79b1c206 100644
--- a/versioned_docs/version-2.1/practical-guide/log-storage-analysis.md
+++ b/versioned_docs/version-2.1/practical-guide/log-storage-analysis.md
@@ -86,26 +86,26 @@ Here is a typical example of a semi-structured log in JSON
format. The top-level
```JSON
{
-"timestamp": "2014-03-08T00:50:03.8432810Z",
-"source": "ADOPTIONCUSTOMERS81",
-"node": "Engine000000000405",
-"level": "Information",
-"component": "DOWNLOADER",
-"clientRequestId": "671db15d-abad-94f6-dd93-b3a2e6000672",
-"message": "Downloading file path:
benchmark/2014/ADOPTIONCUSTOMERS81_94_0.parquet.gz",
-"properties": {
-"size": 1495636750,
-"format": "parquet",
-"rowCount": 855138,
-"downloadDuration": "00:01:58.3520561"
-}
+ "timestamp": "2014-03-08T00:50:03.8432810Z",
+ "source": "ADOPTIONCUSTOMERS81",
+ "node": "Engine000000000405",
+ "level": "Information",
+ "component": "DOWNLOADER",
+ "clientRequestId": "671db15d-abad-94f6-dd93-b3a2e6000672",
+ "message": "Downloading file path:
benchmark/2014/ADOPTIONCUSTOMERS81_94_0.parquet.gz",
+ "properties": {
+ "size": 1495636750,
+ "format": "parquet",
+ "rowCount": 855138,
+ "downloadDuration": "00:01:58.3520561"
+ }
}
```
Apache Doris provides several aspects of support for Flexible Schema log data:
-- For minor changes to top-level fields, Light Schema Change can be used to
add or remove columns and to add or remove indexes, enabling schema changes to
be completed in seconds. When planning a log platform, users only need to
consider which fields need to be indexed.
+- For changes to top-level fields, Light Schema Change can be used to add or
remove columns and to add or remove indexes, enabling schema changes to be
completed in seconds. When planning a log platform, users only need to consider
which fields need to be indexed.
- For extension fields similar to properties, the native semi-structured data
type `VARIANT` is provided, which can write any JSON data, automatically
recognize field names and types in JSON, and automatically split frequently
occurring fields for columnar storage for subsequent analysis. Additionally,
`VARIANT` can create inverted indexes to accelerate internal field queries and
retrievals.
@@ -125,17 +125,17 @@ Before deploying the cluster, you need to estimate the
hardware resources requir
1. Estimate the resources for data writing by the following calculation
formulas:
-- Daily data increment / 86400 s = Average write throughput
+- `Average write throughput = Daily data increment / 86400 s`
-- Average write throughput \* Ratio of the peak write throughput to the
average write throughput = Peak write throughput
+- `Peak write throughput = Average write throughput \* Ratio of the peak write
throughput to the average write throughput`
-- Peak write throughput / Write throughput of a single-core CPU = Number of
CPU cores for the peak write throughput
+- `Number of CPU cores for the peak write throughput = Peak write throughput /
Write throughput of a single-core CPU`
-2. Estimate the resources for data storage by the calculation formula: Daily
data increment / Data compression ratio \* Number of data copies \* Data
storage duration = Data storage space.
+1. Estimate the resources for data storage by the calculation formula:
`Storage space = Daily data increment / Data compression ratio * Number of data
copies * Data storage duration`.
-3. Estimate the resources for data querying. The resources for data querying
depend on the query volume and complexity. It is recommended to reserve 50% of
CPU resources for data query initially and then adjust according to the actual
test results.
+2. Estimate the resources for data querying. The resources for data querying
depend on the query volume and complexity. It is recommended to reserve 50% of
CPU resources for data query initially and then adjust according to the actual
test results.
-4. Integrate the calculation results as follows:
+3. Integrate the calculation results as follows:
1. Divide the number of CPU cores calculated in Step 1 and Step 3 by the
number of CPU cores of a BE server, and you can get the number of BE servers.
@@ -143,39 +143,39 @@ Before deploying the cluster, you need to estimate the
hardware resources requir
3. Allocate the storage space required for each BE server to 4 to 12 data
disks, and you can get the storage capacity required for a single data disk.
-For example, suppose that the daily data increment is 100 TB, the data
compression ratio is 7, the number of data copies is 1, the storage duration of
hot data is 3 days, the storage duration of cold data is 30 days, the ratio of
the peak write throughput to the average write throughput is 200%, the write
throughput of a single-core CUP is 10 MB/s, and 50% of CPU resources are
reserved for data querying, one can estimate that:
+For example, suppose that the daily data increment is 100 TB, the data
compression ratio is 5, the number of data copies is 1, the storage duration of
hot data is 3 days, the storage duration of cold data is 30 days, the ratio of
the peak write throughput to the average write throughput is 200%, the write
throughput of a single-core CUP is 10 MB/s, and 50% of CPU resources are
reserved for data querying, one can estimate that:
-- 3 FE servers are required, each configured with a 16-core CPU, 64 GB memory,
and an 1100 GB SSD disk.
+- 3 FE servers are required, each configured with a 16-core CPU, 64 GB memory,
and an 1 100 GB SSD disk.
-- 15 BE servers are required, each configured with a 32-core CPU, 256 GB
memory, and 8 500 GB SSD disks.
+- 15 BE servers are required, each configured with a 32-core CPU, 256 GB
memory, and 10 600 GB SSD disks.
-- S3:430 TB
+- S3 object storage space 600 TB
Refer to the following table to learn about the values of indicators in the
example above and how they are calculated.
| Indicator (Unit) | Value | Description |
| --- | --- | --- |
| Daily data increment (TB) | 100 | Specify the value according to your actual
needs. |
-| Data compression ratio | 7 | Specify the value according to your actual
needs, which is often between 5 to 10. Note that the data contains index data. |
+| Data compression ratio | 5 | Specify the value according to your actual
needs, which is often between 3 to 10. Note that the data contains index data. |
| Number of data copies | 1 | Specify the value according to your actual
needs, which can be 1, 2, or 3. The default value is 1. |
| Storage duration of hot data (day) | 3 | Specify the value according to
your actual needs. |
| Storage duration of cold data (day) | 30 | Specify the value according to
your actual needs. |
-| Data storage duration | 33 | Calculation formula: Storage duration of hot
data + Storage duration of cold data |
-| Estimated storage space for hot data (TB) | 42.9 | Calculation formula:
Daily data increment / Data compression ratios \* Number of data copies \*
Storage duration of hot data |
-| Estimated storage space for cold data (TB) | 428.6 | Calculation formula:
Daily data increment / Data compression ratios \* Number of data copies \*
Storage duration of cold data |
+| Data storage duration | 33 | Calculation formula: `Storage duration of hot
data + Storage duration of cold data` |
+| Estimated storage space for hot data (TB) | 60 | Calculation formula: `Daily
data increment / Data compression ratios * Number of data copies * Storage
duration of hot data` |
+| Estimated storage space for cold data (TB) | 600 | Calculation formula:
`Daily data increment / Data compression ratios * Number of data copies *
Storage duration of cold data` |
| Ratio of the peak write throughput to the average write throughput | 200% |
Specify the value according to your actual needs. The default value is 200%. |
| Number of CPU cores of a BE server | 32 | Specify the value according to
your actual needs. The default value is 32. |
-| Average write throughput (MB/s) | 1214 | Calculation formula: Daily data
increment / 86400 s |
-| Peak write throughput (MB/s) | 2427 | Calculation formula: Average write
throughput \* Ratio of the peak write throughput to the average write
throughput |
-| Number of CPU cores for the peak write throughput | 242.7 | Calculation
formula: Peak write throughput / Write throughput of a single-core CPU |
+| Average write throughput (MB/s) | 1214 | Calculation formula: `Daily data
increment / 86400 s` |
+| Peak write throughput (MB/s) | 2427 | Calculation formula: `Average write
throughput * Ratio of the peak write throughput to the average write
throughput` |
+| Number of CPU cores for the peak write throughput | 242.7 | Calculation
formula: `Peak write throughput / Write throughput of a single-core CPU` |
| Percent of CPU resources reserved for data querying | 50% | Specify the
value according to your actual needs. The default value is 50%. |
-| Estimated number of BE servers | 15.2 | Calculation formula: Number of CPU
cores for the peak write throughput / Number of CPU cores of a BE server /（1 -
Percent of CPU resources reserved for data querying） |
-| Rounded number of BE servers | 15 | Calculation formula: MAX (Number of
data copies, Estimated number of BE servers) |
-| Estimated data storage space for each BE server (TB) | 4.03 | Calculation
formula: Estimated storage space for hot data / Estimated number of BE servers
/（1 - 30%）, where 30% represents the percent of reserved storage space.<br
/><br />It is recommended to mount 4 to 12 data disks on each BE server to
enhance I/O capabilities. |
+| Estimated number of BE servers | 15.2 | Calculation formula: `Number of CPU
cores for the peak write throughput / Number of CPU cores of a BE server /(1 -
Percent of CPU resources reserved for data querying)` |
+| Rounded number of BE servers | 15 | Calculation formula: `MAX (Number of
data copies, Estimated number of BE servers)` |
+| Estimated data storage space for each BE server (TB) | 5.7 | Calculation
formula: `Estimated storage space for hot data / Estimated number of BE servers
/(1 - 30%)`, where 30% represents the percent of reserved storage space.<br
/><br />It is recommended to mount 4 to 12 data disks on each BE server to
enhance I/O capabilities. |
### Step 2: Deploy the cluster
-After estimating the resources, you need to deploy the cluster. It is
recommended to deploy in both physical and virtual environments manually. For
manual deployment, refer to [Manual
Deployment](../install/cluster-deployment/standard-deployment.md).
+After estimating the resources, you need to deploy the cluster. It is
recommended to deploy in both physical and virtual environments manually. For
manual deployment, refer to [Manual
Deployment](../../install/cluster-deployment/standard-deployment.md).
### Step 3: Optimize FE and BE configurations
@@ -191,11 +191,10 @@ You can find FE configuration fields in
`fe/conf/fe.conf`. Refer to the followin
| `streaming_label_keep_max_second = 3600` `label_keep_max_second = 7200` |
Increase the retention time to handle high-frequency import transactions with
high memory usage. |
| `enable_round_robin_create_tablet = true` | When creating
Tablets, use a Round Robin strategy to distribute evenly. |
| `tablet_rebalancer_type = partition` | When
balancing Tablets, use a strategy to evenly distribute within each partition. |
-| `enable_single_replica_load = true` | Enable
single-replica import, where multiple replicas only need to build an index once
to reduce CPU consumption. |
| `autobucket_min_buckets = 10` | Increase the
minimum number of automatically bucketed buckets from 1 to 10 to avoid
insufficient buckets when the log volume increases. |
| `max_backend_heartbeat_failure_tolerance_count = 10` | In log
scenarios, the BE server may experience high pressure, leading to short-term
timeouts, so increase the tolerance count from 1 to 10. |
-For more information, refer to [FE
Configuration](../admin-manual/config/fe-config.md).
+For more information, refer to [FE
Configuration](../../admin-manual/config/fe-config.md).
**Optimize BE configurations**
@@ -208,7 +207,6 @@ You can find BE configuration fields in `be/conf/be.conf`.
Refer to the followin
| - | `file_cache_path = [{"path": "/mnt/datadisk0/file_cache",
"total_size":53687091200, "query_limit": "10737418240"},{"path":
"/mnt/datadisk1/file_cache", "total_size":53687091200,"query_limit":
"10737418240"}]` | Configure the cache path and related settings for cold data
with the following specific configurations:<br/>`path`: cache
path<br/>`total_size`: total size of the cache path in bytes, where 53687091200
bytes equals 50 GB<br/>`query_limit`: maximum amount of data tha [...]
| Write | `write_buffer_size = 1073741824` |
Increase the file size of the write buffer to reduce small files and random I/O
operations, improving performance. |
| - | `max_tablet_version_num = 20000` |
In coordination with the time_series compaction strategy for table creation,
allow more versions to remain temporarily unmerged |
-| - | `enable_single_replica_load = true` |
Enable single replica import to reduce CPU consumption, consistent with FE
configuration. |
| Compaction | `max_cumu_compaction_threads = 8` |
Set to CPU core count / 4, indicating that 1/4 of CPU resources are used for
writing, 1/4 for background compaction, and 2/1 for queries and other
operations. |
| - | `inverted_index_compaction_enable = true` |
Enable inverted index compaction to reduce CPU consumption during compaction. |
| - | `enable_segcompaction = false` `enable_ordered_data_compaction
= false` | Disable two compaction features that are unnecessary for log
scenarios. |
@@ -225,7 +223,7 @@ You can find BE configuration fields in `be/conf/be.conf`.
Refer to the followin
| - | `trash_file_expire_time_sec = 300`
`path_gc_check_interval_second = 900` `path_scan_interval_second = 900` |
Accelerate the recycling of trash files. |
-For more information, refer to [BE
Configuration](../admin-manual/config/be-config.md).
+For more information, refer to [BE
Configuration](../../admin-manual/config/be-config.md).
### Step 4: Create tables
@@ -235,41 +233,43 @@ Due to the distinct characteristics of both writing and
querying log data, it is
- For data partitioning:
- - Enable [range
partitioning](https://doris.apache.org/docs/table-design/data-partition#range-partition)
with [dynamic
partitions](https://doris.apache.org/docs/table-design/data-partition#dynamic-partition)
managed automatically by day.
+ - Enable [range
partitioning](../../table-design/data-partition#range-partition) (`PARTITION BY
RANGE(`ts`)`) with [dynamic
partitions](../../table-design/data-partition#dynamic-partition)
(`"dynamic_partition.enable" = "true"`) managed automatically by day.
- - Use a field in the DATETIME type as the key for accelerated retrieval of
the latest N log entries.
+ - Use a field in the DATETIME type as the key (`DUPLICATE KEY(ts)`) for
accelerated retrieval of the latest N log entries.
- For data bucketing:
- - Configure the number of buckets to be roughly three times the total
number of disks in the cluster.
+ - Configure the number of buckets to be roughly three times the total
number of disks in the cluster, with each bucket containing approximately 5GB
of data after compression.
- - Use the Random strategy to optimize batch writing efficiency when paired
with single tablet imports.
+ - Use the Random strategy (`DISTRIBUTED BY RANDOM BUCKETS 60`) to optimize
batch writing efficiency when paired with single tablet imports.
-For more information, refer to [Data
Partitioning](../table-design/data-partition.md).
+For more information, refer to [Data
Partitioning](../../table-design/data-partition.md).
-**Configure compaction fileds**
+**Configure compression parameters**
-Configure compaction fields as follows:
+Use the zstd compression algorithm ("compression" = "zstd") to improve data
compression efficiency.
-- Use the time_series strategy to reduce write amplification, which is crucial
for high-throughput log writes.
+**Configure compaction parameters**
+
+Configure compaction fields as follows:
-- Use single-replica compaction to minimize the overhead of multi-replica
compaction.
+- Use the time_series strategy (`"compaction_policy" = "time_series"`) to
reduce write amplification, which is crucial for high-throughput log writes.
-**Configure index fields**
+**Configure index parameters**
Configuring index fields as follows:
-- Create indexes for fields that are frequently queried.
+- Create indexes for fields that are frequently queried (`USING INVERTED`).
- For fields that require full-text search, specify the parser field as
unicode, which satisfies most requirements. If there is a need to support
phrase queries, set the support_phrase field to true; if not needed, set it to
false to reduce storage space.
-**Configure storage policies**
+**Configure storage parameters**
Configure storage policies as follows:
-- For storage of hot data, if using cloud storage, configure the number of
data copies as 1; if using physical disks, configure the number of data copies
as at least 2.
+- For storage of hot data, if using cloud storage, configure the number of
data copies as 1; if using physical disks, configure the number of data copies
as at least 2 (`"replication_num" = "2"`).
-- Configure the storage location for log_s3 and set the log_policy_3day
policy, where the data is cooled and moved to the specified storage location of
log_s3 after 3 days. Refer to the code below.
+- Configure the storage location for log_s3 (`CREATE RESOURCE "log_s3"`) and
set the log_policy_3day policy (`CREATE STORAGE POLICY log_policy_3day`), where
the data is cooled and moved to the specified storage location of log_s3 after
3 days. Refer to the code below.
```SQL
CREATE DATABASE log_db;
@@ -306,20 +306,20 @@ CREATE TABLE log_table
ENGINE = OLAP
DUPLICATE KEY(`ts`)
PARTITION BY RANGE(`ts`) ()
-DISTRIBUTED BY RANDOM BUCKETS 250
+DISTRIBUTED BY RANDOM BUCKETS 60
PROPERTIES (
-"compaction_policy" = "time_series",
-"dynamic_partition.enable" = "true",
-"dynamic_partition.create_history_partition" = "true",
-"dynamic_partition.time_unit" = "DAY",
-"dynamic_partition.start" = "-30",
-"dynamic_partition.end" = "1",
-"dynamic_partition.prefix" = "p",
-"dynamic_partition.buckets" = "250",
-"dynamic_partition.replication_num" = "2", -- unneccessary for the
compute-storage coupled mode
-"replication_num" = "2" -- unneccessary for the compute-storage coupled mode
-"enable_single_replica_compaction" = "true", -- unneccessary for the
compute-storage coupled mode
-"storage_policy" = "log_policy_3day" -- unneccessary for the compute-storage
coupled mode
+ "compression" = "zstd",
+ "compaction_policy" = "time_series",
+ "dynamic_partition.enable" = "true",
+ "dynamic_partition.create_history_partition" = "true",
+ "dynamic_partition.time_unit" = "DAY",
+ "dynamic_partition.start" = "-30",
+ "dynamic_partition.end" = "1",
+ "dynamic_partition.prefix" = "p",
+ "dynamic_partition.buckets" = "60",
+ "dynamic_partition.replication_num" = "2", -- unneccessary for the
compute-storage coupled mode
+ "replication_num" = "2", -- unneccessary for the compute-storage coupled mode
+ "storage_policy" = "log_policy_3day" -- unneccessary for the compute-storage
coupled mode
);
```
@@ -396,7 +396,7 @@ output {
./bin/logstash -f logstash_demo.conf
```
-For more information about the Logstash Doris Output plugin, see [Logstash
Doris Output Plugin](../ecosystem/logstash.md).
+For more information about the Logstash Doris Output plugin, see [Logstash
Doris Output Plugin](../../ecosystem/logstash.md).
**Integrating Filebeat**
@@ -464,7 +464,7 @@ headers:
./filebeat-doris-1.0.0 -c filebeat_demo.yml
```
-For more information about Filebeat, refer to [Beats Doris Output
Plugin](../ecosystem/beats.md).
+For more information about Filebeat, refer to [Beats Doris Output
Plugin](../../ecosystem/beats.md).
**Integrating Kafka**
@@ -497,7 +497,7 @@ FROM KAFKA (
<br />SHOW ROUTINE LOAD;
```
-For more information about Kafka, see [Routine
Load](../data-operate/import/routine-load-manual.md)。
+For more information about Kafka, see [Routine
Load](../../data-operate/import/import-way/routine-load-manual.md).
**Using customized programs to collect logs**
diff --git a/versioned_docs/version-3.0/practical-guide/log-storage-analysis.md
b/versioned_docs/version-3.0/practical-guide/log-storage-analysis.md
index f000b40bf25..22b79b1c206 100644
--- a/versioned_docs/version-3.0/practical-guide/log-storage-analysis.md
+++ b/versioned_docs/version-3.0/practical-guide/log-storage-analysis.md
@@ -86,26 +86,26 @@ Here is a typical example of a semi-structured log in JSON
format. The top-level
```JSON
{
-"timestamp": "2014-03-08T00:50:03.8432810Z",
-"source": "ADOPTIONCUSTOMERS81",
-"node": "Engine000000000405",
-"level": "Information",
-"component": "DOWNLOADER",
-"clientRequestId": "671db15d-abad-94f6-dd93-b3a2e6000672",
-"message": "Downloading file path:
benchmark/2014/ADOPTIONCUSTOMERS81_94_0.parquet.gz",
-"properties": {
-"size": 1495636750,
-"format": "parquet",
-"rowCount": 855138,
-"downloadDuration": "00:01:58.3520561"
-}
+ "timestamp": "2014-03-08T00:50:03.8432810Z",
+ "source": "ADOPTIONCUSTOMERS81",
+ "node": "Engine000000000405",
+ "level": "Information",
+ "component": "DOWNLOADER",
+ "clientRequestId": "671db15d-abad-94f6-dd93-b3a2e6000672",
+ "message": "Downloading file path:
benchmark/2014/ADOPTIONCUSTOMERS81_94_0.parquet.gz",
+ "properties": {
+ "size": 1495636750,
+ "format": "parquet",
+ "rowCount": 855138,
+ "downloadDuration": "00:01:58.3520561"
+ }
}
```
Apache Doris provides several aspects of support for Flexible Schema log data:
-- For minor changes to top-level fields, Light Schema Change can be used to
add or remove columns and to add or remove indexes, enabling schema changes to
be completed in seconds. When planning a log platform, users only need to
consider which fields need to be indexed.
+- For changes to top-level fields, Light Schema Change can be used to add or
remove columns and to add or remove indexes, enabling schema changes to be
completed in seconds. When planning a log platform, users only need to consider
which fields need to be indexed.
- For extension fields similar to properties, the native semi-structured data
type `VARIANT` is provided, which can write any JSON data, automatically
recognize field names and types in JSON, and automatically split frequently
occurring fields for columnar storage for subsequent analysis. Additionally,
`VARIANT` can create inverted indexes to accelerate internal field queries and
retrievals.
@@ -125,17 +125,17 @@ Before deploying the cluster, you need to estimate the
hardware resources requir
1. Estimate the resources for data writing by the following calculation
formulas:
-- Daily data increment / 86400 s = Average write throughput
+- `Average write throughput = Daily data increment / 86400 s`
-- Average write throughput \* Ratio of the peak write throughput to the
average write throughput = Peak write throughput
+- `Peak write throughput = Average write throughput \* Ratio of the peak write
throughput to the average write throughput`
-- Peak write throughput / Write throughput of a single-core CPU = Number of
CPU cores for the peak write throughput
+- `Number of CPU cores for the peak write throughput = Peak write throughput /
Write throughput of a single-core CPU`
-2. Estimate the resources for data storage by the calculation formula: Daily
data increment / Data compression ratio \* Number of data copies \* Data
storage duration = Data storage space.
+1. Estimate the resources for data storage by the calculation formula:
`Storage space = Daily data increment / Data compression ratio * Number of data
copies * Data storage duration`.
-3. Estimate the resources for data querying. The resources for data querying
depend on the query volume and complexity. It is recommended to reserve 50% of
CPU resources for data query initially and then adjust according to the actual
test results.
+2. Estimate the resources for data querying. The resources for data querying
depend on the query volume and complexity. It is recommended to reserve 50% of
CPU resources for data query initially and then adjust according to the actual
test results.
-4. Integrate the calculation results as follows:
+3. Integrate the calculation results as follows:
1. Divide the number of CPU cores calculated in Step 1 and Step 3 by the
number of CPU cores of a BE server, and you can get the number of BE servers.
@@ -143,39 +143,39 @@ Before deploying the cluster, you need to estimate the
hardware resources requir
3. Allocate the storage space required for each BE server to 4 to 12 data
disks, and you can get the storage capacity required for a single data disk.
-For example, suppose that the daily data increment is 100 TB, the data
compression ratio is 7, the number of data copies is 1, the storage duration of
hot data is 3 days, the storage duration of cold data is 30 days, the ratio of
the peak write throughput to the average write throughput is 200%, the write
throughput of a single-core CUP is 10 MB/s, and 50% of CPU resources are
reserved for data querying, one can estimate that:
+For example, suppose that the daily data increment is 100 TB, the data
compression ratio is 5, the number of data copies is 1, the storage duration of
hot data is 3 days, the storage duration of cold data is 30 days, the ratio of
the peak write throughput to the average write throughput is 200%, the write
throughput of a single-core CUP is 10 MB/s, and 50% of CPU resources are
reserved for data querying, one can estimate that:
-- 3 FE servers are required, each configured with a 16-core CPU, 64 GB memory,
and an 1100 GB SSD disk.
+- 3 FE servers are required, each configured with a 16-core CPU, 64 GB memory,
and an 1 100 GB SSD disk.
-- 15 BE servers are required, each configured with a 32-core CPU, 256 GB
memory, and 8 500 GB SSD disks.
+- 15 BE servers are required, each configured with a 32-core CPU, 256 GB
memory, and 10 600 GB SSD disks.
-- S3:430 TB
+- S3 object storage space 600 TB
Refer to the following table to learn about the values of indicators in the
example above and how they are calculated.
| Indicator (Unit) | Value | Description |
| --- | --- | --- |
| Daily data increment (TB) | 100 | Specify the value according to your actual
needs. |
-| Data compression ratio | 7 | Specify the value according to your actual
needs, which is often between 5 to 10. Note that the data contains index data. |
+| Data compression ratio | 5 | Specify the value according to your actual
needs, which is often between 3 to 10. Note that the data contains index data. |
| Number of data copies | 1 | Specify the value according to your actual
needs, which can be 1, 2, or 3. The default value is 1. |
| Storage duration of hot data (day) | 3 | Specify the value according to
your actual needs. |
| Storage duration of cold data (day) | 30 | Specify the value according to
your actual needs. |
-| Data storage duration | 33 | Calculation formula: Storage duration of hot
data + Storage duration of cold data |
-| Estimated storage space for hot data (TB) | 42.9 | Calculation formula:
Daily data increment / Data compression ratios \* Number of data copies \*
Storage duration of hot data |
-| Estimated storage space for cold data (TB) | 428.6 | Calculation formula:
Daily data increment / Data compression ratios \* Number of data copies \*
Storage duration of cold data |
+| Data storage duration | 33 | Calculation formula: `Storage duration of hot
data + Storage duration of cold data` |
+| Estimated storage space for hot data (TB) | 60 | Calculation formula: `Daily
data increment / Data compression ratios * Number of data copies * Storage
duration of hot data` |
+| Estimated storage space for cold data (TB) | 600 | Calculation formula:
`Daily data increment / Data compression ratios * Number of data copies *
Storage duration of cold data` |
| Ratio of the peak write throughput to the average write throughput | 200% |
Specify the value according to your actual needs. The default value is 200%. |
| Number of CPU cores of a BE server | 32 | Specify the value according to
your actual needs. The default value is 32. |
-| Average write throughput (MB/s) | 1214 | Calculation formula: Daily data
increment / 86400 s |
-| Peak write throughput (MB/s) | 2427 | Calculation formula: Average write
throughput \* Ratio of the peak write throughput to the average write
throughput |
-| Number of CPU cores for the peak write throughput | 242.7 | Calculation
formula: Peak write throughput / Write throughput of a single-core CPU |
+| Average write throughput (MB/s) | 1214 | Calculation formula: `Daily data
increment / 86400 s` |
+| Peak write throughput (MB/s) | 2427 | Calculation formula: `Average write
throughput * Ratio of the peak write throughput to the average write
throughput` |
+| Number of CPU cores for the peak write throughput | 242.7 | Calculation
formula: `Peak write throughput / Write throughput of a single-core CPU` |
| Percent of CPU resources reserved for data querying | 50% | Specify the
value according to your actual needs. The default value is 50%. |
-| Estimated number of BE servers | 15.2 | Calculation formula: Number of CPU
cores for the peak write throughput / Number of CPU cores of a BE server /（1 -
Percent of CPU resources reserved for data querying） |
-| Rounded number of BE servers | 15 | Calculation formula: MAX (Number of
data copies, Estimated number of BE servers) |
-| Estimated data storage space for each BE server (TB) | 4.03 | Calculation
formula: Estimated storage space for hot data / Estimated number of BE servers
/（1 - 30%）, where 30% represents the percent of reserved storage space.<br
/><br />It is recommended to mount 4 to 12 data disks on each BE server to
enhance I/O capabilities. |
+| Estimated number of BE servers | 15.2 | Calculation formula: `Number of CPU
cores for the peak write throughput / Number of CPU cores of a BE server /(1 -
Percent of CPU resources reserved for data querying)` |
+| Rounded number of BE servers | 15 | Calculation formula: `MAX (Number of
data copies, Estimated number of BE servers)` |
+| Estimated data storage space for each BE server (TB) | 5.7 | Calculation
formula: `Estimated storage space for hot data / Estimated number of BE servers
/(1 - 30%)`, where 30% represents the percent of reserved storage space.<br
/><br />It is recommended to mount 4 to 12 data disks on each BE server to
enhance I/O capabilities. |
### Step 2: Deploy the cluster
-After estimating the resources, you need to deploy the cluster. It is
recommended to deploy in both physical and virtual environments manually. For
manual deployment, refer to [Manual
Deployment](../install/cluster-deployment/standard-deployment.md).
+After estimating the resources, you need to deploy the cluster. It is
recommended to deploy in both physical and virtual environments manually. For
manual deployment, refer to [Manual
Deployment](../../install/cluster-deployment/standard-deployment.md).
### Step 3: Optimize FE and BE configurations
@@ -191,11 +191,10 @@ You can find FE configuration fields in
`fe/conf/fe.conf`. Refer to the followin
| `streaming_label_keep_max_second = 3600` `label_keep_max_second = 7200` |
Increase the retention time to handle high-frequency import transactions with
high memory usage. |
| `enable_round_robin_create_tablet = true` | When creating
Tablets, use a Round Robin strategy to distribute evenly. |
| `tablet_rebalancer_type = partition` | When
balancing Tablets, use a strategy to evenly distribute within each partition. |
-| `enable_single_replica_load = true` | Enable
single-replica import, where multiple replicas only need to build an index once
to reduce CPU consumption. |
| `autobucket_min_buckets = 10` | Increase the
minimum number of automatically bucketed buckets from 1 to 10 to avoid
insufficient buckets when the log volume increases. |
| `max_backend_heartbeat_failure_tolerance_count = 10` | In log
scenarios, the BE server may experience high pressure, leading to short-term
timeouts, so increase the tolerance count from 1 to 10. |
-For more information, refer to [FE
Configuration](../admin-manual/config/fe-config.md).
+For more information, refer to [FE
Configuration](../../admin-manual/config/fe-config.md).
**Optimize BE configurations**
@@ -208,7 +207,6 @@ You can find BE configuration fields in `be/conf/be.conf`.
Refer to the followin
| - | `file_cache_path = [{"path": "/mnt/datadisk0/file_cache",
"total_size":53687091200, "query_limit": "10737418240"},{"path":
"/mnt/datadisk1/file_cache", "total_size":53687091200,"query_limit":
"10737418240"}]` | Configure the cache path and related settings for cold data
with the following specific configurations:<br/>`path`: cache
path<br/>`total_size`: total size of the cache path in bytes, where 53687091200
bytes equals 50 GB<br/>`query_limit`: maximum amount of data tha [...]
| Write | `write_buffer_size = 1073741824` |
Increase the file size of the write buffer to reduce small files and random I/O
operations, improving performance. |
| - | `max_tablet_version_num = 20000` |
In coordination with the time_series compaction strategy for table creation,
allow more versions to remain temporarily unmerged |
-| - | `enable_single_replica_load = true` |
Enable single replica import to reduce CPU consumption, consistent with FE
configuration. |
| Compaction | `max_cumu_compaction_threads = 8` |
Set to CPU core count / 4, indicating that 1/4 of CPU resources are used for
writing, 1/4 for background compaction, and 2/1 for queries and other
operations. |
| - | `inverted_index_compaction_enable = true` |
Enable inverted index compaction to reduce CPU consumption during compaction. |
| - | `enable_segcompaction = false` `enable_ordered_data_compaction
= false` | Disable two compaction features that are unnecessary for log
scenarios. |
@@ -225,7 +223,7 @@ You can find BE configuration fields in `be/conf/be.conf`.
Refer to the followin
| - | `trash_file_expire_time_sec = 300`
`path_gc_check_interval_second = 900` `path_scan_interval_second = 900` |
Accelerate the recycling of trash files. |
-For more information, refer to [BE
Configuration](../admin-manual/config/be-config.md).
+For more information, refer to [BE
Configuration](../../admin-manual/config/be-config.md).
### Step 4: Create tables
@@ -235,41 +233,43 @@ Due to the distinct characteristics of both writing and
querying log data, it is
- For data partitioning:
- - Enable [range
partitioning](https://doris.apache.org/docs/table-design/data-partition#range-partition)
with [dynamic
partitions](https://doris.apache.org/docs/table-design/data-partition#dynamic-partition)
managed automatically by day.
+ - Enable [range
partitioning](../../table-design/data-partition#range-partition) (`PARTITION BY
RANGE(`ts`)`) with [dynamic
partitions](../../table-design/data-partition#dynamic-partition)
(`"dynamic_partition.enable" = "true"`) managed automatically by day.
- - Use a field in the DATETIME type as the key for accelerated retrieval of
the latest N log entries.
+ - Use a field in the DATETIME type as the key (`DUPLICATE KEY(ts)`) for
accelerated retrieval of the latest N log entries.
- For data bucketing:
- - Configure the number of buckets to be roughly three times the total
number of disks in the cluster.
+ - Configure the number of buckets to be roughly three times the total
number of disks in the cluster, with each bucket containing approximately 5GB
of data after compression.
- - Use the Random strategy to optimize batch writing efficiency when paired
with single tablet imports.
+ - Use the Random strategy (`DISTRIBUTED BY RANDOM BUCKETS 60`) to optimize
batch writing efficiency when paired with single tablet imports.
-For more information, refer to [Data
Partitioning](../table-design/data-partition.md).
+For more information, refer to [Data
Partitioning](../../table-design/data-partition.md).
-**Configure compaction fileds**
+**Configure compression parameters**
-Configure compaction fields as follows:
+Use the zstd compression algorithm ("compression" = "zstd") to improve data
compression efficiency.
-- Use the time_series strategy to reduce write amplification, which is crucial
for high-throughput log writes.
+**Configure compaction parameters**
+
+Configure compaction fields as follows:
-- Use single-replica compaction to minimize the overhead of multi-replica
compaction.
+- Use the time_series strategy (`"compaction_policy" = "time_series"`) to
reduce write amplification, which is crucial for high-throughput log writes.
-**Configure index fields**
+**Configure index parameters**
Configuring index fields as follows:
-- Create indexes for fields that are frequently queried.
+- Create indexes for fields that are frequently queried (`USING INVERTED`).
- For fields that require full-text search, specify the parser field as
unicode, which satisfies most requirements. If there is a need to support
phrase queries, set the support_phrase field to true; if not needed, set it to
false to reduce storage space.
-**Configure storage policies**
+**Configure storage parameters**
Configure storage policies as follows:
-- For storage of hot data, if using cloud storage, configure the number of
data copies as 1; if using physical disks, configure the number of data copies
as at least 2.
+- For storage of hot data, if using cloud storage, configure the number of
data copies as 1; if using physical disks, configure the number of data copies
as at least 2 (`"replication_num" = "2"`).
-- Configure the storage location for log_s3 and set the log_policy_3day
policy, where the data is cooled and moved to the specified storage location of
log_s3 after 3 days. Refer to the code below.
+- Configure the storage location for log_s3 (`CREATE RESOURCE "log_s3"`) and
set the log_policy_3day policy (`CREATE STORAGE POLICY log_policy_3day`), where
the data is cooled and moved to the specified storage location of log_s3 after
3 days. Refer to the code below.
```SQL
CREATE DATABASE log_db;
@@ -306,20 +306,20 @@ CREATE TABLE log_table
ENGINE = OLAP
DUPLICATE KEY(`ts`)
PARTITION BY RANGE(`ts`) ()
-DISTRIBUTED BY RANDOM BUCKETS 250
+DISTRIBUTED BY RANDOM BUCKETS 60
PROPERTIES (
-"compaction_policy" = "time_series",
-"dynamic_partition.enable" = "true",
-"dynamic_partition.create_history_partition" = "true",
-"dynamic_partition.time_unit" = "DAY",
-"dynamic_partition.start" = "-30",
-"dynamic_partition.end" = "1",
-"dynamic_partition.prefix" = "p",
-"dynamic_partition.buckets" = "250",
-"dynamic_partition.replication_num" = "2", -- unneccessary for the
compute-storage coupled mode
-"replication_num" = "2" -- unneccessary for the compute-storage coupled mode
-"enable_single_replica_compaction" = "true", -- unneccessary for the
compute-storage coupled mode
-"storage_policy" = "log_policy_3day" -- unneccessary for the compute-storage
coupled mode
+ "compression" = "zstd",
+ "compaction_policy" = "time_series",
+ "dynamic_partition.enable" = "true",
+ "dynamic_partition.create_history_partition" = "true",
+ "dynamic_partition.time_unit" = "DAY",
+ "dynamic_partition.start" = "-30",
+ "dynamic_partition.end" = "1",
+ "dynamic_partition.prefix" = "p",
+ "dynamic_partition.buckets" = "60",
+ "dynamic_partition.replication_num" = "2", -- unneccessary for the
compute-storage coupled mode
+ "replication_num" = "2", -- unneccessary for the compute-storage coupled mode
+ "storage_policy" = "log_policy_3day" -- unneccessary for the compute-storage
coupled mode
);
```
@@ -396,7 +396,7 @@ output {
./bin/logstash -f logstash_demo.conf
```
-For more information about the Logstash Doris Output plugin, see [Logstash
Doris Output Plugin](../ecosystem/logstash.md).
+For more information about the Logstash Doris Output plugin, see [Logstash
Doris Output Plugin](../../ecosystem/logstash.md).
**Integrating Filebeat**
@@ -464,7 +464,7 @@ headers:
./filebeat-doris-1.0.0 -c filebeat_demo.yml
```
-For more information about Filebeat, refer to [Beats Doris Output
Plugin](../ecosystem/beats.md).
+For more information about Filebeat, refer to [Beats Doris Output
Plugin](../../ecosystem/beats.md).
**Integrating Kafka**
@@ -497,7 +497,7 @@ FROM KAFKA (
<br />SHOW ROUTINE LOAD;
```
-For more information about Kafka, see [Routine
Load](../data-operate/import/routine-load-manual.md)。
+For more information about Kafka, see [Routine
Load](../../data-operate/import/import-way/routine-load-manual.md).
**Using customized programs to collect logs**
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
