This is an automated email from the ASF dual-hosted git repository.
qiaojialin pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iotdb-docs.git
The following commit(s) were added to refs/heads/main by this push:
new 4cff14ad Add device level TTL docs (#399)
4cff14ad is described below
commit 4cff14ad37961a82236332d5ce7432f0023078ba
Author: shuwenwei <[email protected]>
AuthorDate: Fri Nov 8 16:37:03 2024 +0800
Add device level TTL docs (#399)
---
.../Master/Reference/Common-Config-Manual.md | 29 ++++++++++
src/UserGuide/Master/stage/Delete-Data/TTL.md | 61 ++++++++++++++++------
.../Master/Reference/Common-Config-Manual.md | 37 +++++++++----
src/zh/UserGuide/Master/stage/Delete-Data/TTL.md | 56 +++++++++++++++-----
4 files changed, 144 insertions(+), 39 deletions(-)
diff --git a/src/UserGuide/Master/Reference/Common-Config-Manual.md
b/src/UserGuide/Master/Reference/Common-Config-Manual.md
index 3d1f0ce3..6eb96364 100644
--- a/src/UserGuide/Master/Reference/Common-Config-Manual.md
+++ b/src/UserGuide/Master/Reference/Common-Config-Manual.md
@@ -636,6 +636,35 @@ Different configuration parameters take effect in the
following three ways:
|Default| 100000 |
|Effective|After restarting system|
+### TTL 配置
+* ttl\_check\_interval
+
+| Name | ttl\_check\_interval
|
+|:-----------:|:-------------------------------------------------------------------------------|
+| Description | The interval of TTL check task in each database. Unit: ms.
Default is 2 hours. |
+| Type | int
|
+| Default | 7200000
|
+| Effective | After restarting system
|
+
+* max\_expired\_time
+
+| Name | max\_expired\_time
|
+| :----------:
|:--------------------------------------------------------------------------------------------------------------------------------------------------|
+| Description | If a file contains device that has expired for more
than this duration, then the file will be settled immediately. Unit: ms.
Default is 1 month. |
+| Type | int
|
+| Default | 2592000000
|
+| Effective | After restarting system
|
+
+* expired\_data\_ratio
+
+| Name | expired\_data\_ratio
|
+| :----------:
|:--------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Description | The expired device ratio. If the ratio of expired
devices in one file exceeds this value, then expired data of this file will be
cleaned by compaction. |
+| Type | float
|
+| Default | 0.3
|
+| Effective | After restarting system
|
+
+
### Storage Engine Configuration
* timestamp\_precision
diff --git a/src/UserGuide/Master/stage/Delete-Data/TTL.md
b/src/UserGuide/Master/stage/Delete-Data/TTL.md
index 18be915d..66270809 100644
--- a/src/UserGuide/Master/stage/Delete-Data/TTL.md
+++ b/src/UserGuide/Master/stage/Delete-Data/TTL.md
@@ -21,29 +21,47 @@
# TTL
-IoTDB supports storage-level TTL settings, which means it is able to delete
old data automatically and periodically. The benefit of using TTL is that
hopefully you can control the total disk space usage and prevent the machine
from running out of disks. Moreover, the query performance may downgrade as the
total number of files goes up and the memory usage also increase as there are
more files. Timely removing such files helps to keep at a high query
performance level and reduce memory usage.
+IoTDB supports device-level TTL settings, which means it is able to delete old
data automatically and periodically. The benefit of using TTL is that hopefully
you can control the total disk space usage and prevent the machine from running
out of disks. Moreover, the query performance may downgrade as the total number
of files goes up and the memory usage also increases as there are more files.
Timely removing such files helps to keep at a high query performance level and
reduce memory usage.
The default unit of TTL is milliseconds. If the time precision in the
configuration file changes to another, the TTL is still set to milliseconds.
-## Set TTL
-
-The SQL Statement for setting TTL is as follow:
-
-```
-IoTDB> set ttl to root.ln 3600000
-```
-
-This example means that for data in `root.ln`, only 3600000 ms, that is, the
latest 1 hour will remain, the older one is removed or made invisible.
+When setting TTL, the system will look for all devices included in the set
path and set TTL for these devices. The system will delete expired data at the
device granularity.
+After the device data expires, it will not be queryable. The data in the disk
file cannot be guaranteed to be deleted immediately, but it can be guaranteed
to be deleted eventually.
+However, due to operational costs, the expired data will not be physically
deleted right after expiring. The physical deletion is delayed until compaction.
+Therefore, before the data is physically deleted, if the TTL is reduced or
lifted, it may cause data that was previously invisible due to TTL to reappear.
+The system can only set up to 1000 TTL rules, and when this limit is reached,
some TTL rules need to be deleted before new rules can be set.
+## TTL Path Rule
+The path can only be prefix paths (i.e., the path cannot contain \* , except
\*\* in the last level).
+This path will match devices and also allows users to specify paths without
asterisks as specific databases or devices.
+When the path does not contain asterisks, the system will check if it matches
a database; if it matches a database, both the path and path.\*\* will be set
at the same time. Note: Device TTL settings do not verify the existence of
metadata, i.e., it is allowed to set TTL for a non-existent device.
```
-IoTDB> set ttl to root.sgcc.** 3600000
+qualified paths：
+root.**
+root.db.**
+root.db.group1.**
+root.db
+root.db.group1.d1
+
+unqualified paths：
+root.*.db
+root.**.db.*
+root.db.*
```
-It supports setting TTL for databases in a path. This example represents
setting TTL for all databases in the `root.sgcc` path.
+## TTL Applicable Rules
+When a device is subject to multiple TTL rules, the more precise and longer
rules are prioritized. For example, for the device
"root.bj.hd.dist001.turbine001", the rule "root.bj.hd.dist001.turbine001" takes
precedence over "root.bj.hd.dist001.\*\*", and the rule
"root.bj.hd.dist001.\*\*" takes precedence over "root.bj.hd.**".
+## Set TTL
+The set ttl operation can be understood as setting a TTL rule, for example,
setting ttl to root.sg.group1.** is equivalent to mounting ttl for all devices
that can match this path pattern.
+The unset ttl operation indicates unmounting TTL for the corresponding path
pattern; if there is no corresponding TTL, nothing will be done.
+If you want to set TTL to be infinitely large, you can use the INF keyword.
+The SQL Statement for setting TTL is as follow:
```
-IoTDB> set ttl to root.** 3600000
+set ttl to pathPattern 360000;
```
-This example represents setting TTL for all databases.
-
+Set the Time to Live (TTL) to a pathPattern of 360,000 milliseconds; the
pathPattern should not contain a wildcard (\*) in the middle and must end with
a double asterisk (\*\*). The pathPattern is used to match corresponding
devices.
+To maintain compatibility with older SQL syntax, if the user-provided
pathPattern matches a database (db), the path pattern is automatically expanded
to include all sub-paths denoted by path.\*\*.
+For instance, writing "set ttl to root.sg 360000" will automatically be
transformed into "set ttl to root.sg.\*\* 360000", which sets the TTL for all
devices under root.sg. However, if the specified pathPattern does not match a
database, the aforementioned logic will not apply. For example, writing "set
ttl to root.sg.group 360000" will not be expanded to "root.sg.group.\*\*" since
root.sg.group does not match a database.
+It is also permissible to specify a particular device without a wildcard (*).
## Unset TTL
To unset TTL, we can use follwing SQL statement:
@@ -87,4 +105,15 @@ IoTDB> show all ttl
|root.sgcc| null|
| root.DB|3600000|
+----------+-------+
-```
\ No newline at end of file
+```
+Display devices' ttl
+```
+IoTDB> show devices
++---------------+---------+---------+
+| Device|IsAligned| TTL|
++---------------+---------+---------+
+|root.sg.device1| false| 36000000|
+|root.sg.device2| true| INF|
++---------------+---------+---------+
+```
+All devices will definitely have a TTL, meaning it cannot be null. INF
represents infinity.
\ No newline at end of file
diff --git a/src/zh/UserGuide/Master/Reference/Common-Config-Manual.md
b/src/zh/UserGuide/Master/Reference/Common-Config-Manual.md
index 1632e11b..eee800cd 100644
--- a/src/zh/UserGuide/Master/Reference/Common-Config-Manual.md
+++ b/src/zh/UserGuide/Master/Reference/Common-Config-Manual.md
@@ -666,6 +666,34 @@ IoTDB ConfigNode 和 DataNode 的公共配置参数位于 `conf` 目录下。
|默认值| 100000 |
|改后生效方式|重启生效|
+#### TTL 配置
+* ttl\_check\_interval
+
+| 名字 | ttl\_check\_interval |
+| :----------: |:-------------------------|
+| 描述 | ttl 检查任务的间隔，单位 ms，默认为 2h |
+| 类型 | int |
+| 默认值 | 7200000 |
+| 改后生效方式 | 重启生效 |
+
+* max\_expired\_time
+
+| 名字 | max\_expired\_time |
+| :----------: |:-----------------------------|
+| 描述 | 如果一个文件中存在设备已经过期超过此时间，那么这个文件将被立即整理。单位 ms，默认为一个月 |
+| 类型 | int |
+| 默认值 | 2592000000 |
+| 改后生效方式 | 重启生效 |
+
+* expired\_data\_ratio
+
+| 名字 | expired\_data\_ratio |
+| :----------: |:----------------------------------------------------------|
+| 描述 | 过期设备比例。如果一个文件中过期设备的比率超过这个值，那么这个文件中的过期数据将通过 compaction 清理。 |
+| 类型 | float |
+| 默认值 | 0.3 |
+| 改后生效方式 | 重启生效 |
+
#### 存储引擎配置
* timestamp\_precision
@@ -677,15 +705,6 @@ IoTDB ConfigNode 和 DataNode 的公共配置参数位于 `conf` 目录下。
| 默认值 | ms |
| 改后生效方式 | 仅允许在第一次启动服务前修改 |
-* default\_ttl\_in\_ms
-
-| 名字 | default\_ttl\_in\_ms |
-| :----------: | :---------------------------------------------------------- |
-| 描述 | 数据保留时间，会丢弃 now()-default\_ttl 之前的数据，单位 ms |
-| 类型 | long |
-| 默认值 | 36000000 |
-| 改后生效方式 | 重启生效 |
-
* max\_waiting\_time\_when\_insert\_blocked
| 名字 | max\_waiting\_time\_when\_insert\_blocked |
diff --git a/src/zh/UserGuide/Master/stage/Delete-Data/TTL.md
b/src/zh/UserGuide/Master/stage/Delete-Data/TTL.md
index 64da492e..04792669 100644
--- a/src/zh/UserGuide/Master/stage/Delete-Data/TTL.md
+++ b/src/zh/UserGuide/Master/stage/Delete-Data/TTL.md
@@ -21,28 +21,46 @@
# 数据存活时间（TTL）
-IoTDB 支持对 database 级别设置数据存活时间（TTL），这使得 IoTDB 可以定期、自动地删除一定时间之前的数据。合理使用 TTL
+IoTDB 支持对 device 级别设置数据存活时间（TTL），这使得 IoTDB 可以定期、自动地删除一定时间之前的数据。合理使用 TTL
可以帮助您控制 IoTDB 占用的总磁盘空间以避免出现磁盘写满等异常。并且，随着文件数量的增多，查询性能往往随之下降，
内存占用也会有所提高。及时地删除一些较老的文件有助于使查询性能维持在一个较高的水平和减少内存资源的占用。
TTL的默认单位为毫秒，如果配置文件中的时间精度修改为其他单位，设置ttl时仍然使用毫秒单位。
+当设置 TTL 时，系统会根据设置的路径寻找所包含的所有 device，并为这些 device 设置 TTL 时间，系统会按设备粒度对过期数据进行删除。
+当设备数据过期后，将不能被查询到，但磁盘文件中的数据不能保证立即删除（会在一定时间内删除），但可以保证最终被删除。
+考虑到操作代价，系统不会立即物理删除超过 TTL 的数据，而是通过合并来延迟地物理删除。因此，在数据被物理删除前，如果调小或者解除 TTL，可能会导致之前因
TTL 而不可见的数据重新出现。
+系统中仅能设置至多 1000 条 TTL 规则，达到该上限时，需要先删除部分 TTL 规则才能设置新的规则
+
+## TTL Path 规则
+设置的路径 path 只支持前缀路径（即路径中间不能带 \* ， 且必须以 \*\* 结尾），该路径会匹配到设备，也允许用户指定不带星的 path 为具体的
database 或 device，当 path 不带 \* 时，会检查是否匹配到 database，若匹配到 database，则会同时设置 path 和
path.\*\*。
+注意：设备 TTL 设置不会对元数据的存在性进行校验，即允许对一条不存在的设备设置 TTL。
+```
+合格的 path：
+root.**
+root.db.**
+root.db.group1.**
+root.db
+root.db.group1.d1
+
+不合格的 path：
+root.*.db
+root.**.db.*
+root.db.*
+```
+## TTL 适用规则
+当一个设备适用多条TTL规则时，优先适用较精确和较长的规则。例如对于设备“root.bj.hd.dist001.turbine001”来说，规则“root.bj.hd.dist001.turbine001”比“root.bj.hd.dist001.\*\*”优先，而规则“root.bj.hd.dist001.\*\*”比“root.bj.hd.\*\*”优先；
## 设置 TTL
-
+set ttl 操作可以理解为设置一条 TTL规则，比如 set ttl to root.sg.group1.\*\*
就相当于对所有可以匹配到该路径模式的设备挂载 ttl。 unset ttl 操作表示对相应路径模式卸载 TTL，若不存在对应 TTL，则不做任何事。若想把
TTL 调成无限大，则可以使用 INF 关键字
设置 TTL 的 SQL 语句如下所示：
```
-IoTDB> set ttl to root.ln 3600000
-```
-这个例子表示在`root.ln`数据库中，只有3600000毫秒，即最近一个小时的数据将会保存，旧数据会被移除或不可见。
-```
-IoTDB> set ttl to root.sgcc.** 3600000
-```
-支持给某一路径下的 database 设置TTL，这个例子表示`root.sgcc`路径下的所有 database 设置TTL。
-```
-IoTDB> set ttl to root.** 3600000
+set ttl to pathPattern 360000;
```
-表示给所有 database 设置TTL。
-
+pathPattern 是前缀路径，即路径中间不能带 \* 且必须以 \*\* 结尾。
+pathPattern 匹配对应的设备。为了兼容老版本 SQL 语法，允许用户输入的 pathPattern 匹配到 db，则自动将前缀路径扩展为
path.\*\*。
+例如，写set ttl to root.sg 360000 则会自动转化为set ttl to root.sg.\*\* 360000，转化后的语句对所有
root.sg 下的 device 设置TTL。
+但若写的 pathPattern 无法匹配到 db，则上述逻辑不会生效。
+如写set ttl to root.sg.group 360000 ，由于root.sg.group未匹配到
db，则不会被扩充为root.sg.group.\*\*。 也允许指定具体 device，不带 \*。
## 取消 TTL
取消 TTL 的 SQL 语句如下所示：
@@ -86,4 +104,14 @@ IoTDB> show all ttl
| root.DB|3600000|
+-------------+-------+
```
-
+显示设备的 TTL。
+```
+IoTDB> show devices
++---------------+---------+---------+
+| Device|IsAligned| TTL|
++---------------+---------+---------+
+|root.sg.device1| false| 36000000|
+|root.sg.device2| true| INF|
++---------------+---------+---------+
+```
+所有设备都一定会有 TTL，即不可能是 null。INF 表示无穷大。
